[jira] [Resolved] (SPARK-37690) Recursive view `df` detected (cycle: `df` -> `df`)
[
https://issues.apache.org/jira/browse/SPARK-37690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Imran Rashid resolved SPARK-37690.
--
Fix Version/s: 3.2.2
3.3.0
Resolution: Duplicate
> Recursive view `df` detected (cycle: `df` -> `df`)
> --
>
> Key: SPARK-37690
> URL: https://issues.apache.org/jira/browse/SPARK-37690
> Project: Spark
> Issue Type: Bug
> Components: PySpark
>Affects Versions: 3.2.0
>Reporter: Robin
>Priority: Major
> Fix For: 3.2.2, 3.3.0
>
>
> In Spark 3.2.0, you can no longer reuse the same name for a temporary view.
> This change is backwards incompatible, and means a common way of running
> pipelines of SQL queries no longer works. The following is a simple
> reproducible example that works in Spark 2.x and 3.1.2, but not in 3.2.0:
> {code:python}from pyspark.context import SparkContext
> from pyspark.sql import SparkSession
> sc = SparkContext.getOrCreate()
> spark = SparkSession(sc)
> sql = """ SELECT id as col_1, rand() AS col_2 FROM RANGE(10); """
> df = spark.sql(sql)
> df.createOrReplaceTempView("df")
> sql = """ SELECT * FROM df """
> df = spark.sql(sql)
> df.createOrReplaceTempView("df")
> sql = """ SELECT * FROM df """
> df = spark.sql(sql) {code}
> The following error is now produced:
> {code:python}AnalysisException: Recursive view `df` detected (cycle: `df` ->
> `df`)
> {code}
> I'm reasonably sure this change is unintentional in 3.2.0 since it breaks a
> lot of legacy code, and the `createOrReplaceTempView` method is named
> explicitly such that replacing an existing view should be allowed. An
> internet search suggests other users have run into a similar problems, e.g.
> [here|https://community.databricks.com/s/question/0D53f1Qugr7CAB/upgrading-from-spark-24-to-32-recursive-view-errors-when-using]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-32003) Shuffle files for lost executor are not unregistered if fetch failure occurs after executor is lost
[ https://issues.apache.org/jira/browse/SPARK-32003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-32003: - Fix Version/s: 2.4.7 > Shuffle files for lost executor are not unregistered if fetch failure occurs > after executor is lost > --- > > Key: SPARK-32003 > URL: https://issues.apache.org/jira/browse/SPARK-32003 > Project: Spark > Issue Type: Bug > Components: Scheduler >Affects Versions: 2.4.6, 3.0.0 >Reporter: Wing Yew Poon >Assignee: Wing Yew Poon >Priority: Major > Fix For: 2.4.7, 3.0.1, 3.1.0 > > > A customer's cluster has a node that goes down while a Spark application is > running. (They are running Spark on YARN with the external shuffle service > enabled.) An executor is lost (apparently the only one running on the node). > This executor lost event is handled in the DAGScheduler, which removes the > executor from its BlockManagerMaster. At this point, there is no > unregistering of shuffle files for the executor or the node. Soon after, > tasks trying to fetch shuffle files output by that executor fail with > FetchFailed (because the node is down, there is no NodeManager available to > serve shuffle files). By right, such fetch failures should cause the shuffle > files for the executor to be unregistered, but they do not. > Due to task failure, the stage is re-attempted. Tasks continue to fail due to > fetch failure form the lost executor's shuffle output. This time, since the > failed epoch for the executor is higher, the executor is removed again (this > doesn't really do anything, the executor was already removed when it was > lost) and this time the shuffle output is unregistered. > So it takes two stage attempts instead of one to clear the shuffle output. We > get 4 attempts by default. The customer was unlucky and two nodes went down > during the stage, i.e., the same problem happened twice. So they used up 4 > stage attempts and the stage failed and thus the job. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-32003) Shuffle files for lost executor are not unregistered if fetch failure occurs after executor is lost
[ https://issues.apache.org/jira/browse/SPARK-32003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17171083#comment-17171083 ] Imran Rashid commented on SPARK-32003: -- Fixed in 2.4.7 by https://github.com/apache/spark/pull/29182 > Shuffle files for lost executor are not unregistered if fetch failure occurs > after executor is lost > --- > > Key: SPARK-32003 > URL: https://issues.apache.org/jira/browse/SPARK-32003 > Project: Spark > Issue Type: Bug > Components: Scheduler >Affects Versions: 2.4.6, 3.0.0 >Reporter: Wing Yew Poon >Assignee: Wing Yew Poon >Priority: Major > Fix For: 2.4.7, 3.0.1, 3.1.0 > > > A customer's cluster has a node that goes down while a Spark application is > running. (They are running Spark on YARN with the external shuffle service > enabled.) An executor is lost (apparently the only one running on the node). > This executor lost event is handled in the DAGScheduler, which removes the > executor from its BlockManagerMaster. At this point, there is no > unregistering of shuffle files for the executor or the node. Soon after, > tasks trying to fetch shuffle files output by that executor fail with > FetchFailed (because the node is down, there is no NodeManager available to > serve shuffle files). By right, such fetch failures should cause the shuffle > files for the executor to be unregistered, but they do not. > Due to task failure, the stage is re-attempted. Tasks continue to fail due to > fetch failure form the lost executor's shuffle output. This time, since the > failed epoch for the executor is higher, the executor is removed again (this > doesn't really do anything, the executor was already removed when it was > lost) and this time the shuffle output is unregistered. > So it takes two stage attempts instead of one to clear the shuffle output. We > get 4 attempts by default. The customer was unlucky and two nodes went down > during the stage, i.e., the same problem happened twice. So they used up 4 > stage attempts and the stage failed and thus the job. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-32003) Shuffle files for lost executor are not unregistered if fetch failure occurs after executor is lost
[ https://issues.apache.org/jira/browse/SPARK-32003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17170945#comment-17170945 ] Imran Rashid commented on SPARK-32003: -- Fixed in 3.0.1 by https://github.com/apache/spark/pull/29193 > Shuffle files for lost executor are not unregistered if fetch failure occurs > after executor is lost > --- > > Key: SPARK-32003 > URL: https://issues.apache.org/jira/browse/SPARK-32003 > Project: Spark > Issue Type: Bug > Components: Scheduler >Affects Versions: 2.4.6, 3.0.0 >Reporter: Wing Yew Poon >Assignee: Wing Yew Poon >Priority: Major > Fix For: 3.0.1, 3.1.0 > > > A customer's cluster has a node that goes down while a Spark application is > running. (They are running Spark on YARN with the external shuffle service > enabled.) An executor is lost (apparently the only one running on the node). > This executor lost event is handled in the DAGScheduler, which removes the > executor from its BlockManagerMaster. At this point, there is no > unregistering of shuffle files for the executor or the node. Soon after, > tasks trying to fetch shuffle files output by that executor fail with > FetchFailed (because the node is down, there is no NodeManager available to > serve shuffle files). By right, such fetch failures should cause the shuffle > files for the executor to be unregistered, but they do not. > Due to task failure, the stage is re-attempted. Tasks continue to fail due to > fetch failure form the lost executor's shuffle output. This time, since the > failed epoch for the executor is higher, the executor is removed again (this > doesn't really do anything, the executor was already removed when it was > lost) and this time the shuffle output is unregistered. > So it takes two stage attempts instead of one to clear the shuffle output. We > get 4 attempts by default. The customer was unlucky and two nodes went down > during the stage, i.e., the same problem happened twice. So they used up 4 > stage attempts and the stage failed and thus the job. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-32003) Shuffle files for lost executor are not unregistered if fetch failure occurs after executor is lost
[ https://issues.apache.org/jira/browse/SPARK-32003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-32003: - Fix Version/s: 3.0.1 > Shuffle files for lost executor are not unregistered if fetch failure occurs > after executor is lost > --- > > Key: SPARK-32003 > URL: https://issues.apache.org/jira/browse/SPARK-32003 > Project: Spark > Issue Type: Bug > Components: Scheduler >Affects Versions: 2.4.6, 3.0.0 >Reporter: Wing Yew Poon >Assignee: Wing Yew Poon >Priority: Major > Fix For: 3.0.1, 3.1.0 > > > A customer's cluster has a node that goes down while a Spark application is > running. (They are running Spark on YARN with the external shuffle service > enabled.) An executor is lost (apparently the only one running on the node). > This executor lost event is handled in the DAGScheduler, which removes the > executor from its BlockManagerMaster. At this point, there is no > unregistering of shuffle files for the executor or the node. Soon after, > tasks trying to fetch shuffle files output by that executor fail with > FetchFailed (because the node is down, there is no NodeManager available to > serve shuffle files). By right, such fetch failures should cause the shuffle > files for the executor to be unregistered, but they do not. > Due to task failure, the stage is re-attempted. Tasks continue to fail due to > fetch failure form the lost executor's shuffle output. This time, since the > failed epoch for the executor is higher, the executor is removed again (this > doesn't really do anything, the executor was already removed when it was > lost) and this time the shuffle output is unregistered. > So it takes two stage attempts instead of one to clear the shuffle output. We > get 4 attempts by default. The customer was unlucky and two nodes went down > during the stage, i.e., the same problem happened twice. So they used up 4 > stage attempts and the stage failed and thus the job. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-32003) Shuffle files for lost executor are not unregistered if fetch failure occurs after executor is lost
[ https://issues.apache.org/jira/browse/SPARK-32003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-32003. -- Fix Version/s: 3.1.0 Resolution: Fixed Fixed in master for 3.1 by https://github.com/apache/spark/pull/28848 > Shuffle files for lost executor are not unregistered if fetch failure occurs > after executor is lost > --- > > Key: SPARK-32003 > URL: https://issues.apache.org/jira/browse/SPARK-32003 > Project: Spark > Issue Type: Bug > Components: Scheduler >Affects Versions: 2.4.6, 3.0.0 >Reporter: Wing Yew Poon >Assignee: Wing Yew Poon >Priority: Major > Fix For: 3.1.0 > > > A customer's cluster has a node that goes down while a Spark application is > running. (They are running Spark on YARN with the external shuffle service > enabled.) An executor is lost (apparently the only one running on the node). > This executor lost event is handled in the DAGScheduler, which removes the > executor from its BlockManagerMaster. At this point, there is no > unregistering of shuffle files for the executor or the node. Soon after, > tasks trying to fetch shuffle files output by that executor fail with > FetchFailed (because the node is down, there is no NodeManager available to > serve shuffle files). By right, such fetch failures should cause the shuffle > files for the executor to be unregistered, but they do not. > Due to task failure, the stage is re-attempted. Tasks continue to fail due to > fetch failure form the lost executor's shuffle output. This time, since the > failed epoch for the executor is higher, the executor is removed again (this > doesn't really do anything, the executor was already removed when it was > lost) and this time the shuffle output is unregistered. > So it takes two stage attempts instead of one to clear the shuffle output. We > get 4 attempts by default. The customer was unlucky and two nodes went down > during the stage, i.e., the same problem happened twice. So they used up 4 > stage attempts and the stage failed and thus the job. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-32003) Shuffle files for lost executor are not unregistered if fetch failure occurs after executor is lost
[ https://issues.apache.org/jira/browse/SPARK-32003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reassigned SPARK-32003: Assignee: Wing Yew Poon > Shuffle files for lost executor are not unregistered if fetch failure occurs > after executor is lost > --- > > Key: SPARK-32003 > URL: https://issues.apache.org/jira/browse/SPARK-32003 > Project: Spark > Issue Type: Bug > Components: Scheduler >Affects Versions: 2.4.6, 3.0.0 >Reporter: Wing Yew Poon >Assignee: Wing Yew Poon >Priority: Major > > A customer's cluster has a node that goes down while a Spark application is > running. (They are running Spark on YARN with the external shuffle service > enabled.) An executor is lost (apparently the only one running on the node). > This executor lost event is handled in the DAGScheduler, which removes the > executor from its BlockManagerMaster. At this point, there is no > unregistering of shuffle files for the executor or the node. Soon after, > tasks trying to fetch shuffle files output by that executor fail with > FetchFailed (because the node is down, there is no NodeManager available to > serve shuffle files). By right, such fetch failures should cause the shuffle > files for the executor to be unregistered, but they do not. > Due to task failure, the stage is re-attempted. Tasks continue to fail due to > fetch failure form the lost executor's shuffle output. This time, since the > failed epoch for the executor is higher, the executor is removed again (this > doesn't really do anything, the executor was already removed when it was > lost) and this time the shuffle output is unregistered. > So it takes two stage attempts instead of one to clear the shuffle output. We > get 4 attempts by default. The customer was unlucky and two nodes went down > during the stage, i.e., the same problem happened twice. So they used up 4 > stage attempts and the stage failed and thus the job. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-25561) HiveClient.getPartitionsByFilter throws an exception if Hive retries directSql
[ https://issues.apache.org/jira/browse/SPARK-25561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-25561: - Description: In HiveShim.scala, the current behahttps://issues.apache.org/jira/browse/SPARK-25561#vior is that if hive.metastore.try.direct.sql is enabled, we expect the getPartitionsByFilter call to succeed. If it fails, we'll throw a RuntimeException. However, this might not always be the case. Hive's direct SQL functionality is best-attempt. Meaning, it will fall back to ORM if direct sql fails. Spark should handle that exception correctly if Hive falls back to ORM. was: In HiveShim.scala, the current behavior is that if hive.metastore.try.direct.sql is enabled, we expect the getPartitionsByFilter call to succeed. If it fails, we'll throw a RuntimeException. However, this might not always be the case. Hive's direct SQL functionality is best-attempt. Meaning, it will fall back to ORM if direct sql fails. Spark should handle that exception correctly if Hive falls back to ORM. > HiveClient.getPartitionsByFilter throws an exception if Hive retries directSql > -- > > Key: SPARK-25561 > URL: https://issues.apache.org/jira/browse/SPARK-25561 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.1.0 >Reporter: Karthik Manamcheri >Priority: Major > > In HiveShim.scala, the current > behahttps://issues.apache.org/jira/browse/SPARK-25561#vior is that if > hive.metastore.try.direct.sql is enabled, we expect the getPartitionsByFilter > call to succeed. If it fails, we'll throw a RuntimeException. > However, this might not always be the case. Hive's direct SQL functionality > is best-attempt. Meaning, it will fall back to ORM if direct sql fails. Spark > should handle that exception correctly if Hive falls back to ORM. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-25561) HiveClient.getPartitionsByFilter throws an exception if Hive retries directSql
[ https://issues.apache.org/jira/browse/SPARK-25561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-25561: - Description: In HiveShim.scala, the current behavior is that if hive.metastore.try.direct.sql is enabled, we expect the getPartitionsByFilter call to succeed. If it fails, we'll throw a RuntimeException. However, this might not always be the case. Hive's direct SQL functionality is best-attempt. Meaning, it will fall back to ORM if direct sql fails. Spark should handle that exception correctly if Hive falls back to ORM. was: In HiveShim.scala, the current behahttps://issues.apache.org/jira/browse/SPARK-25561#vior is that if hive.metastore.try.direct.sql is enabled, we expect the getPartitionsByFilter call to succeed. If it fails, we'll throw a RuntimeException. However, this might not always be the case. Hive's direct SQL functionality is best-attempt. Meaning, it will fall back to ORM if direct sql fails. Spark should handle that exception correctly if Hive falls back to ORM. > HiveClient.getPartitionsByFilter throws an exception if Hive retries directSql > -- > > Key: SPARK-25561 > URL: https://issues.apache.org/jira/browse/SPARK-25561 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.1.0 >Reporter: Karthik Manamcheri >Priority: Major > > In HiveShim.scala, the current behavior is that if > hive.metastore.try.direct.sql is enabled, we expect the getPartitionsByFilter > call to succeed. If it fails, we'll throw a RuntimeException. > However, this might not always be the case. Hive's direct SQL functionality > is best-attempt. Meaning, it will fall back to ORM if direct sql fails. Spark > should handle that exception correctly if Hive falls back to ORM. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-27324) document configurations related to executor metrics
[ https://issues.apache.org/jira/browse/SPARK-27324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-27324. -- Fix Version/s: 3.0.0 Resolution: Fixed > document configurations related to executor metrics > --- > > Key: SPARK-27324 > URL: https://issues.apache.org/jira/browse/SPARK-27324 > Project: Spark > Issue Type: Improvement > Components: Documentation >Affects Versions: 3.0.0 >Reporter: Wing Yew Poon >Assignee: Wing Yew Poon >Priority: Major > Fix For: 3.0.0 > > > SPARK-23429 introduced executor memory metrics, and the configuration, > spark.eventLog.logStageExecutorMetrics.enabled, that determines if per-stage > per-executor metric peaks get written to the event log. (The metrics are > polled and sent in the heartbeat, and this is always done; the configuration > is only to determine if aggregated metric peaks are written to the event log.) > SPARK-24958 added proc fs based metrics to the executor memory metrics, and > the configuration, spark.eventLog.logStageExecutorProcessTreeMetrics.enabled, > to determine if these additional (more expensive) metrics are collected when > metrics are polled. > SPARK-26329 will introduce a configuration, > spark.executor.metrics.pollingInterval, to allow polling at more frequent > intervals than the executor heartbeat. > These configurations and how they relate to each other should be documented > in the Configuration page. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-27324) document configurations related to executor metrics
[ https://issues.apache.org/jira/browse/SPARK-27324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reassigned SPARK-27324: Assignee: Wing Yew Poon > document configurations related to executor metrics > --- > > Key: SPARK-27324 > URL: https://issues.apache.org/jira/browse/SPARK-27324 > Project: Spark > Issue Type: Improvement > Components: Documentation >Affects Versions: 3.0.0 >Reporter: Wing Yew Poon >Assignee: Wing Yew Poon >Priority: Major > > SPARK-23429 introduced executor memory metrics, and the configuration, > spark.eventLog.logStageExecutorMetrics.enabled, that determines if per-stage > per-executor metric peaks get written to the event log. (The metrics are > polled and sent in the heartbeat, and this is always done; the configuration > is only to determine if aggregated metric peaks are written to the event log.) > SPARK-24958 added proc fs based metrics to the executor memory metrics, and > the configuration, spark.eventLog.logStageExecutorProcessTreeMetrics.enabled, > to determine if these additional (more expensive) metrics are collected when > metrics are polled. > SPARK-26329 will introduce a configuration, > spark.executor.metrics.pollingInterval, to allow polling at more frequent > intervals than the executor heartbeat. > These configurations and how they relate to each other should be documented > in the Configuration page. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27324) document configurations related to executor metrics
[ https://issues.apache.org/jira/browse/SPARK-27324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027807#comment-17027807 ] Imran Rashid commented on SPARK-27324: -- Fixed by https://github.com/apache/spark/pull/27329 > document configurations related to executor metrics > --- > > Key: SPARK-27324 > URL: https://issues.apache.org/jira/browse/SPARK-27324 > Project: Spark > Issue Type: Improvement > Components: Documentation >Affects Versions: 3.0.0 >Reporter: Wing Yew Poon >Assignee: Wing Yew Poon >Priority: Major > > SPARK-23429 introduced executor memory metrics, and the configuration, > spark.eventLog.logStageExecutorMetrics.enabled, that determines if per-stage > per-executor metric peaks get written to the event log. (The metrics are > polled and sent in the heartbeat, and this is always done; the configuration > is only to determine if aggregated metric peaks are written to the event log.) > SPARK-24958 added proc fs based metrics to the executor memory metrics, and > the configuration, spark.eventLog.logStageExecutorProcessTreeMetrics.enabled, > to determine if these additional (more expensive) metrics are collected when > metrics are polled. > SPARK-26329 will introduce a configuration, > spark.executor.metrics.pollingInterval, to allow polling at more frequent > intervals than the executor heartbeat. > These configurations and how they relate to each other should be documented > in the Configuration page. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6235) Address various 2G limits
[ https://issues.apache.org/jira/browse/SPARK-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16998577#comment-16998577 ] Imran Rashid commented on SPARK-6235: - [~sammysheep] Spark's ML library uses the same jira project, if that is what you meant -- but I don't know of what specifically has already been implemented in spark to deal w/ large PCA or if there is another specific issue just for that. I'd suggest you first ask u...@spark.apache.org, since the first question is if there is another way of dealing with this > Address various 2G limits > - > > Key: SPARK-6235 > URL: https://issues.apache.org/jira/browse/SPARK-6235 > Project: Spark > Issue Type: Umbrella > Components: Shuffle, Spark Core >Reporter: Reynold Xin >Priority: Major > Fix For: 2.4.0 > > Attachments: SPARK-6235_Design_V0.02.pdf > > > An umbrella ticket to track the various 2G limit we have in Spark, due to the > use of byte arrays and ByteBuffers. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6235) Address various 2G limits
[ https://issues.apache.org/jira/browse/SPARK-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16997631#comment-16997631 ] Imran Rashid commented on SPARK-6235: - [~sammysheep] are you discussing the use case for task results > 2G? Or large records? Or did you mean one of the parts that was supposed to be fixed in the plan above? I don't deny there is _some_ use for large task result -- I just haven't heard much demand for it (in fact you're the first person I've heard from). Given that, I don't expect to see it fixed immediately. You could open another jira, though honestly for the moment I think it would be more of a place for folks to voice their interest. (I'm pretty sure nothing has changed since 2.4.0 on what is fixed and what is not.) > Address various 2G limits > - > > Key: SPARK-6235 > URL: https://issues.apache.org/jira/browse/SPARK-6235 > Project: Spark > Issue Type: Umbrella > Components: Shuffle, Spark Core >Reporter: Reynold Xin >Priority: Major > Fix For: 2.4.0 > > Attachments: SPARK-6235_Design_V0.02.pdf > > > An umbrella ticket to track the various 2G limit we have in Spark, due to the > use of byte arrays and ByteBuffers. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-27189) Add Executor metrics and memory usage instrumentation to the metrics system
[ https://issues.apache.org/jira/browse/SPARK-27189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-27189. -- Fix Version/s: 3.0.0 Resolution: Fixed Fixed by https://github.com/apache/spark/pull/24132 > Add Executor metrics and memory usage instrumentation to the metrics system > --- > > Key: SPARK-27189 > URL: https://issues.apache.org/jira/browse/SPARK-27189 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 3.0.0 >Reporter: Luca Canali >Assignee: Luca Canali >Priority: Minor > Fix For: 3.0.0 > > Attachments: Example_dashboard_Spark_Memory_Metrics.PNG > > > This proposes to add instrumentation of memory usage via the Spark > Dropwizard/Codahale metrics system. Memory usage metrics are available via > the Executor metrics, recently implemented as detailed in > https://issues.apache.org/jira/browse/SPARK-23206. > Making metrics usage metrics available via the Spark Dropwzard metrics system > allow to improve Spark performance dashboards and study memory usage, as in > the attached example graph. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-27189) Add Executor metrics and memory usage instrumentation to the metrics system
[ https://issues.apache.org/jira/browse/SPARK-27189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reassigned SPARK-27189: Assignee: Luca Canali > Add Executor metrics and memory usage instrumentation to the metrics system > --- > > Key: SPARK-27189 > URL: https://issues.apache.org/jira/browse/SPARK-27189 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 3.0.0 >Reporter: Luca Canali >Assignee: Luca Canali >Priority: Minor > Attachments: Example_dashboard_Spark_Memory_Metrics.PNG > > > This proposes to add instrumentation of memory usage via the Spark > Dropwizard/Codahale metrics system. Memory usage metrics are available via > the Executor metrics, recently implemented as detailed in > https://issues.apache.org/jira/browse/SPARK-23206. > Making metrics usage metrics available via the Spark Dropwzard metrics system > allow to improve Spark performance dashboards and study memory usage, as in > the attached example graph. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-23485) Kubernetes should support node blacklist
[
https://issues.apache.org/jira/browse/SPARK-23485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Imran Rashid updated SPARK-23485:
-
Labels: (was: bulk-closed)
> Kubernetes should support node blacklist
>
>
> Key: SPARK-23485
> URL: https://issues.apache.org/jira/browse/SPARK-23485
> Project: Spark
> Issue Type: New Feature
> Components: Kubernetes, Scheduler
>Affects Versions: 2.4.0, 3.0.0
>Reporter: Imran Rashid
>Priority: Major
>
> Spark's BlacklistTracker maintains a list of "bad nodes" which it will not
> use for running tasks (eg., because of bad hardware). When running in yarn,
> this blacklist is used to avoid ever allocating resources on blacklisted
> nodes:
> https://github.com/apache/spark/blob/e836c27ce011ca9aef822bef6320b4a7059ec343/resource-managers/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala#L128
> I'm just beginning to poke around the kubernetes code, so apologies if this
> is incorrect -- but I didn't see any references to
> {{scheduler.nodeBlacklist()}} in {{KubernetesClusterSchedulerBackend}} so it
> seems this is missing. Thought of this while looking at SPARK-19755, a
> similar issue on mesos.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Reopened] (SPARK-23485) Kubernetes should support node blacklist
[
https://issues.apache.org/jira/browse/SPARK-23485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Imran Rashid reopened SPARK-23485:
--
I think this issue is still valid for current versions, it was just opened
against 2.3.0, so reopening.
> Kubernetes should support node blacklist
>
>
> Key: SPARK-23485
> URL: https://issues.apache.org/jira/browse/SPARK-23485
> Project: Spark
> Issue Type: New Feature
> Components: Kubernetes, Scheduler
>Affects Versions: 2.4.0, 3.0.0
>Reporter: Imran Rashid
>Priority: Major
> Labels: bulk-closed
>
> Spark's BlacklistTracker maintains a list of "bad nodes" which it will not
> use for running tasks (eg., because of bad hardware). When running in yarn,
> this blacklist is used to avoid ever allocating resources on blacklisted
> nodes:
> https://github.com/apache/spark/blob/e836c27ce011ca9aef822bef6320b4a7059ec343/resource-managers/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala#L128
> I'm just beginning to poke around the kubernetes code, so apologies if this
> is incorrect -- but I didn't see any references to
> {{scheduler.nodeBlacklist()}} in {{KubernetesClusterSchedulerBackend}} so it
> seems this is missing. Thought of this while looking at SPARK-19755, a
> similar issue on mesos.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-23485) Kubernetes should support node blacklist
[
https://issues.apache.org/jira/browse/SPARK-23485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Imran Rashid updated SPARK-23485:
-
Affects Version/s: (was: 2.3.0)
3.0.0
2.4.0
> Kubernetes should support node blacklist
>
>
> Key: SPARK-23485
> URL: https://issues.apache.org/jira/browse/SPARK-23485
> Project: Spark
> Issue Type: New Feature
> Components: Kubernetes, Scheduler
>Affects Versions: 2.4.0, 3.0.0
>Reporter: Imran Rashid
>Priority: Major
> Labels: bulk-closed
>
> Spark's BlacklistTracker maintains a list of "bad nodes" which it will not
> use for running tasks (eg., because of bad hardware). When running in yarn,
> this blacklist is used to avoid ever allocating resources on blacklisted
> nodes:
> https://github.com/apache/spark/blob/e836c27ce011ca9aef822bef6320b4a7059ec343/resource-managers/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala#L128
> I'm just beginning to poke around the kubernetes code, so apologies if this
> is incorrect -- but I didn't see any references to
> {{scheduler.nodeBlacklist()}} in {{KubernetesClusterSchedulerBackend}} so it
> seems this is missing. Thought of this while looking at SPARK-19755, a
> similar issue on mesos.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-29762) GPU Scheduling - default task resource amount to 1
[ https://issues.apache.org/jira/browse/SPARK-29762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972703#comment-16972703 ] Imran Rashid commented on SPARK-29762: -- The scenario I was thinking is: application is submitted with app-wide resource requirements for 4 cores & 2 gpus executors, but not for tasks. (I guess they specify "spark.executor.resource.gpu" but not "spark.task.resource.gpu".). One taskset is submitted with no task resource requirements, because it doesn't need any gpus. Another taskset is submitted which does require 1 cpu & 1 gpu per task. The user does it this way because they don't want all of their tasks to use gpus, but they know enough of them will need gpus that they'd rather just request gpus upfront, than scale up & down two different types of executors. > GPU Scheduling - default task resource amount to 1 > -- > > Key: SPARK-29762 > URL: https://issues.apache.org/jira/browse/SPARK-29762 > Project: Spark > Issue Type: Story > Components: Spark Core >Affects Versions: 3.0.0 >Reporter: Thomas Graves >Priority: Major > > Default the task level resource configs (for gpu/fpga, etc) to 1. So if the > user specifies the executor resource then to make it more user friendly lets > have the task resource config default to 1. This is ok right now since we > require resources to have an address. It also matches what we do for the > spark.task.cpus configs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-29762) GPU Scheduling - default task resource amount to 1
[
https://issues.apache.org/jira/browse/SPARK-29762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972631#comment-16972631
]
Imran Rashid commented on SPARK-29762:
--
I don't really understand the complication. I know there would be some special
casing for GPUs in the config parsing code (eg. in
{{org.apache.spark.resource.ResourceUtils#parseResourceRequirements}}), but
doesn't seem anything too bad.
I did think about this more, and realize it gets a bit confusing when you add
in task-level resource constraints. you won't schedule optimally for tasks
that don't need gpu, and you won't have gpus leftover for the tasks that do
need them. Eg, say you had each executor setup with 4 cores and 2 gpus. If
you had one task set come in which only needed cpu, you would only run 2
copies. And then if another taskset came in which did need the gpus, you
woudn't be able to schedule it.
You can't end up in that situation until you have task-specific resource
constraints. But does it get too messy to have sensible defaults in that
situation? Maybe the user specifies gpus as an executor resource up front, for
the whole cluster, because they have them available and they know some
significant fraction of the workloads need them. They might think that the
regular tasks will just ignore the gpus, and the tasks that do need gpus would
just specify them as task-level constraints.
I guess this might have been a bad suggestion after all, sorry.
> GPU Scheduling - default task resource amount to 1
> --
>
> Key: SPARK-29762
> URL: https://issues.apache.org/jira/browse/SPARK-29762
> Project: Spark
> Issue Type: Story
> Components: Spark Core
>Affects Versions: 3.0.0
>Reporter: Thomas Graves
>Priority: Major
>
> Default the task level resource configs (for gpu/fpga, etc) to 1. So if the
> user specifies the executor resource then to make it more user friendly lets
> have the task resource config default to 1. This is ok right now since we
> require resources to have an address. It also matches what we do for the
> spark.task.cpus configs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-22579) BlockManager.getRemoteValues and BlockManager.getRemoteBytes should be implemented using streaming
[
https://issues.apache.org/jira/browse/SPARK-22579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964281#comment-16964281
]
Imran Rashid commented on SPARK-22579:
--
Sorry I had not noticed this issue before. I agree that there is an
inefficiency here, if you did this streaming you could pipeline fetching the
data w/ computing on the data. The existing changes you point to solve the
memory footprint, by fetching to disk, but not actually pipelining the
computation.
That said, this isnt' easy to fix. You need to touch a lot of core stuff in
the network layers, and as you said it gets trickier with handling failures
(you have to throw out all partial work in the current task).
You'll probably still see a discrepancy between runtimes when running locally
vs. remote. Best case, you'd get a 2x speedup with this change. In your use
case, that would still be ~40 seconds to 4 minutes.
> BlockManager.getRemoteValues and BlockManager.getRemoteBytes should be
> implemented using streaming
> --
>
> Key: SPARK-22579
> URL: https://issues.apache.org/jira/browse/SPARK-22579
> Project: Spark
> Issue Type: Improvement
> Components: Block Manager, Spark Core
>Affects Versions: 2.1.0
>Reporter: Eyal Farago
>Priority: Major
>
> when an RDD partition is cached on an executor bu the task requiring it is
> running on another executor (process locality ANY), the cached partition is
> fetched via BlockManager.getRemoteValues which delegates to
> BlockManager.getRemoteBytes, both calls are blocking.
> in my use case I had a 700GB RDD spread over 1000 partitions on a 6 nodes
> cluster, cached to disk. rough math shows that average partition size is
> 700MB.
> looking at spark UI it was obvious that tasks running with process locality
> 'ANY' are much slower than local tasks (~40 seconds to 8-10 minutes ratio), I
> was able to capture thread dumps of executors executing remote tasks and got
> this stake trace:
> {quote}Thread ID Thread Name Thread StateThread Locks
> 1521 Executor task launch worker-1000WAITING
> Lock(java.util.concurrent.ThreadPoolExecutor$Worker@196462978})
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> scala.concurrent.impl.Promise$DefaultPromise.tryAwait(Promise.scala:202)
> scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:218)
> scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
> scala.concurrent.Await$$anonfun$result$1.apply(package.scala:190)
> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
> scala.concurrent.Await$.result(package.scala:190)
> org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:190)
> org.apache.spark.network.BlockTransferService.fetchBlockSync(BlockTransferService.scala:104)
> org.apache.spark.storage.BlockManager.getRemoteBytes(BlockManager.scala:582)
> org.apache.spark.storage.BlockManager.getRemoteValues(BlockManager.scala:550)
> org.apache.spark.storage.BlockManager.get(BlockManager.scala:638)
> org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:690)
> org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:334)
> org.apache.spark.rdd.RDD.iterator(RDD.scala:285)
> org.apache.spark.rdd.ZippedPartitionsRDD2.compute(ZippedPartitionsRDD.scala:89)
> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
> org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
> org.apache.spark.rdd.RDD.iterator(RDD.scala:287){quote}
> digging into the code showed that the block manager first fetches all bytes
> (getRemoteBytes) and then wraps it with a deserialization stream, this has
> several draw backs:
> 1. blocking, requesting executor is blocked while the remote executor is
> serving the block.
> 2. potentially large memory footprint on requesting executor, in my use case
> a 700mb of raw bytes stored in a ChunkedByteBuffer.
> 3. inefficient, requesting side usually don't need all values at once as it
> consumes the values via an iterator.
> 4. potentially large memory footprint on serving executor, in case the block
> is cached in deserialized form the serving executor has to serialize it into
> a ChunkedByteBuffer
[jira] [Resolved] (SPARK-29398) Allow RPC endpoints to use dedicated thread pools
[ https://issues.apache.org/jira/browse/SPARK-29398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-29398. -- Fix Version/s: 3.0.0 Assignee: Marcelo Masiero Vanzin Resolution: Fixed Resolved by https://github.com/apache/spark/pull/26059 > Allow RPC endpoints to use dedicated thread pools > - > > Key: SPARK-29398 > URL: https://issues.apache.org/jira/browse/SPARK-29398 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 3.0.0 >Reporter: Marcelo Masiero Vanzin >Assignee: Marcelo Masiero Vanzin >Priority: Major > Fix For: 3.0.0 > > > This is a new feature of the RPC framework so that we can isolate RPC message > delivery for plugins from normal Spark RPC needs. This minimizes the impact > that plugins can have on normal RPC communication - they'll still fight for > CPU, but they wouldn't block the dispatcher threads used by existing Spark > RPC endpoints. > See parent bug for further details. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-28211) Shuffle Storage API: Driver Lifecycle
[ https://issues.apache.org/jira/browse/SPARK-28211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reassigned SPARK-28211: Assignee: Yifei Huang > Shuffle Storage API: Driver Lifecycle > - > > Key: SPARK-28211 > URL: https://issues.apache.org/jira/browse/SPARK-28211 > Project: Spark > Issue Type: Sub-task > Components: Shuffle >Affects Versions: 3.0.0 >Reporter: Matt Cheah >Assignee: Yifei Huang >Priority: Major > > As part of the shuffle storage API, allow users to hook in application-wide > startup and shutdown methods. This can do things like create tables in the > shuffle storage database, or register / unregister against file servers. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-28211) Shuffle Storage API: Driver Lifecycle
[ https://issues.apache.org/jira/browse/SPARK-28211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-28211. -- Fix Version/s: 3.0.0 Resolution: Fixed Resolved by https://github.com/apache/spark/pull/25823 > Shuffle Storage API: Driver Lifecycle > - > > Key: SPARK-28211 > URL: https://issues.apache.org/jira/browse/SPARK-28211 > Project: Spark > Issue Type: Sub-task > Components: Shuffle >Affects Versions: 3.0.0 >Reporter: Matt Cheah >Assignee: Yifei Huang >Priority: Major > Fix For: 3.0.0 > > > As part of the shuffle storage API, allow users to hook in application-wide > startup and shutdown methods. This can do things like create tables in the > shuffle storage database, or register / unregister against file servers. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-28867) InMemoryStore checkpoint to speed up replay log file in HistoryServer
[ https://issues.apache.org/jira/browse/SPARK-28867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949767#comment-16949767 ] Imran Rashid commented on SPARK-28867: -- This is closely related to SPARK-20656. Its not *quite* a duplicate, because that was about reparsing the logs for the same application within the same SHS instance -- so the SHS still had whatever state stored in memory. Here you're also talking about speeding up parsing of those files even when the SHS is restarted, which also requires some way to restore any state across SHS restarts. > InMemoryStore checkpoint to speed up replay log file in HistoryServer > - > > Key: SPARK-28867 > URL: https://issues.apache.org/jira/browse/SPARK-28867 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 3.0.0 >Reporter: wuyi >Priority: Major > > HistoryServer now could be very slow to replay a large log file at the first > time and it always re-replay an inprogress log file after it changes. we > could periodically checkpoint InMemoryStore to speed up replay log file. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-29396) Extend Spark plugin interface to driver
[
https://issues.apache.org/jira/browse/SPARK-29396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949669#comment-16949669
]
Imran Rashid commented on SPARK-29396:
--
The memory monitor plugin I wrote (why I pushed the executor plugin) was useful
on the driver too -- it just used the hack I mentioned above, of creating a
SparkListener which ignored events.
I think [~lucacanali] also has some use cases involving collecting custom
metrics.
You can write custom code in the driver -- but I think the idea is that this
allows you to turn the plugins on without requiring users to touch their code.
For example, a cluster admin would want a plugin for metric collection, and
they could enable it for users in precompiled jobs, or for users that only
interact via a SQL interface, etc. Similar to how you can call
{{sc.addSparkListener()}} inside your driver program, or you could set the conf
{{spark.extraListeners}}.
> Extend Spark plugin interface to driver
> ---
>
> Key: SPARK-29396
> URL: https://issues.apache.org/jira/browse/SPARK-29396
> Project: Spark
> Issue Type: New Feature
> Components: Spark Core
>Affects Versions: 3.0.0
>Reporter: Marcelo Masiero Vanzin
>Priority: Major
>
> Spark provides an extension API for people to implement executor plugins,
> added in SPARK-24918 and later extended in SPARK-28091.
> That API does not offer any functionality for doing similar things on the
> driver side, though. As a consequence of that, there is not a good way for
> the executor plugins to get information or communicate in any way with the
> Spark driver.
> I've been playing with such an improved API for developing some new
> functionality. I'll file a few child bugs for the work to get the changes in.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-29396) Extend Spark plugin interface to driver
[ https://issues.apache.org/jira/browse/SPARK-29396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948720#comment-16948720 ] Imran Rashid commented on SPARK-29396: -- My hack to get around this in the past was to create a "SparkListener" which just ignored all the events it got, as that lets you instantiate arbitrary code in the driver, after most initialization but before running anything else. Its an ugly api for sure, so it would be nice to improve -- but I'm curious if there is a functional shortcoming you need to address as well? > Extend Spark plugin interface to driver > --- > > Key: SPARK-29396 > URL: https://issues.apache.org/jira/browse/SPARK-29396 > Project: Spark > Issue Type: New Feature > Components: Spark Core >Affects Versions: 3.0.0 >Reporter: Marcelo Masiero Vanzin >Priority: Major > > Spark provides an extension API for people to implement executor plugins, > added in SPARK-24918 and later extended in SPARK-28091. > That API does not offer any functionality for doing similar things on the > driver side, though. As a consequence of that, there is not a good way for > the executor plugins to get information or communicate in any way with the > Spark driver. > I've been playing with such an improved API for developing some new > functionality. I'll file a few child bugs for the work to get the changes in. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-27468) "Storage Level" in "RDD Storage Page" is not correct
[
https://issues.apache.org/jira/browse/SPARK-27468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Imran Rashid resolved SPARK-27468.
--
Fix Version/s: 3.0.0
Resolution: Fixed
Resolved by https://github.com/apache/spark/pull/25779
> "Storage Level" in "RDD Storage Page" is not correct
>
>
> Key: SPARK-27468
> URL: https://issues.apache.org/jira/browse/SPARK-27468
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.4.1, 2.4.2, 2.4.3, 2.4.4, 3.0.0
>Reporter: Shixiong Zhu
>Assignee: Marcelo Masiero Vanzin
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: Screenshot from 2019-04-17 10-42-55.png
>
>
> I ran the following unit test and checked the UI.
> {code}
> val conf = new SparkConf()
> .setAppName("test")
> .setMaster("local-cluster[2,1,1024]")
> .set("spark.ui.enabled", "true")
> sc = new SparkContext(conf)
> val rdd = sc.makeRDD(1 to 10, 1).persist(StorageLevel.MEMORY_ONLY_2)
> rdd.count()
> Thread.sleep(360)
> {code}
> The storage level is "Memory Deserialized 1x Replicated" in the RDD storage
> page.
> I tried to debug and found this is because Spark emitted the following two
> events:
> {code}
> event: SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(1,
> 10.8.132.160, 65473, None),rdd_0_0,StorageLevel(memory, deserialized, 2
> replicas),56,0))
> event: SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(0,
> 10.8.132.160, 65474, None),rdd_0_0,StorageLevel(memory, deserialized, 1
> replicas),56,0))
> {code}
> The storage level in the second event will overwrite the first one. "1
> replicas" comes from this line:
> https://github.com/apache/spark/blob/3ab96d7acf870e53c9016b0b63d0b328eec23bed/core/src/main/scala/org/apache/spark/storage/BlockManager.scala#L1457
> Maybe AppStatusListener should calculate the replicas from events?
> Another fact we may need to think about is when replicas is 2, will two Spark
> events arrive in the same order? Currently, two RPCs from different executors
> can arrive in any order.
> Credit goes to [~srfnmnk] who reported this issue originally.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-27468) "Storage Level" in "RDD Storage Page" is not correct
[
https://issues.apache.org/jira/browse/SPARK-27468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Imran Rashid reassigned SPARK-27468:
Assignee: Marcelo Masiero Vanzin
> "Storage Level" in "RDD Storage Page" is not correct
>
>
> Key: SPARK-27468
> URL: https://issues.apache.org/jira/browse/SPARK-27468
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.4.1, 2.4.2, 2.4.3, 2.4.4, 3.0.0
>Reporter: Shixiong Zhu
>Assignee: Marcelo Masiero Vanzin
>Priority: Major
> Attachments: Screenshot from 2019-04-17 10-42-55.png
>
>
> I ran the following unit test and checked the UI.
> {code}
> val conf = new SparkConf()
> .setAppName("test")
> .setMaster("local-cluster[2,1,1024]")
> .set("spark.ui.enabled", "true")
> sc = new SparkContext(conf)
> val rdd = sc.makeRDD(1 to 10, 1).persist(StorageLevel.MEMORY_ONLY_2)
> rdd.count()
> Thread.sleep(360)
> {code}
> The storage level is "Memory Deserialized 1x Replicated" in the RDD storage
> page.
> I tried to debug and found this is because Spark emitted the following two
> events:
> {code}
> event: SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(1,
> 10.8.132.160, 65473, None),rdd_0_0,StorageLevel(memory, deserialized, 2
> replicas),56,0))
> event: SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(0,
> 10.8.132.160, 65474, None),rdd_0_0,StorageLevel(memory, deserialized, 1
> replicas),56,0))
> {code}
> The storage level in the second event will overwrite the first one. "1
> replicas" comes from this line:
> https://github.com/apache/spark/blob/3ab96d7acf870e53c9016b0b63d0b328eec23bed/core/src/main/scala/org/apache/spark/storage/BlockManager.scala#L1457
> Maybe AppStatusListener should calculate the replicas from events?
> Another fact we may need to think about is when replicas is 2, will two Spark
> events arrive in the same order? Currently, two RPCs from different executors
> can arrive in any order.
> Credit goes to [~srfnmnk] who reported this issue originally.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-29189) Add an option to ignore block locations when listing file
[ https://issues.apache.org/jira/browse/SPARK-29189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16946176#comment-16946176 ] Imran Rashid commented on SPARK-29189: -- Fixed by pr https://github.com/apache/spark/pull/25869 > Add an option to ignore block locations when listing file > - > > Key: SPARK-29189 > URL: https://issues.apache.org/jira/browse/SPARK-29189 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.0.0 >Reporter: Wang, Gang >Assignee: Wang, Gang >Priority: Major > > In our PROD env, we have a pure Spark cluster, I think this is also pretty > common, where computation is separated from storage layer. In such deploy > mode, data locality is never reachable. > And there are some configurations in Spark scheduler to reduce waiting time > for data locality(e.g. "spark.locality.wait"). While, problem is that, in > listing file phase, the location informations of all the files, with all the > blocks inside each file, are all fetched from the distributed file system. > Actually, in a PROD environment, a table can be so huge that even fetching > all these location informations need take tens of seconds. > To improve such scenario, Spark need provide an option, where data locality > can be totally ignored, all we need in the listing file phase are the files > locations, without any block location informations. > > And we made a benchmark in our PROD env, after ignore the block locations, we > got a pretty huge improvement. > ||Table Size||Total File Number||Total Block Number||List File Duration(With > Block Location)||List File Duration(Without Block Location)|| > |22.6T|3|12|16.841s|1.730s| > |28.8 T|42001|148964|10.099s|2.858s| > |3.4 T|2| 2|5.833s|4.881s| > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-29189) Add an option to ignore block locations when listing file
[ https://issues.apache.org/jira/browse/SPARK-29189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reassigned SPARK-29189: Assignee: Wang, Gang > Add an option to ignore block locations when listing file > - > > Key: SPARK-29189 > URL: https://issues.apache.org/jira/browse/SPARK-29189 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.0.0 >Reporter: Wang, Gang >Assignee: Wang, Gang >Priority: Major > > In our PROD env, we have a pure Spark cluster, I think this is also pretty > common, where computation is separated from storage layer. In such deploy > mode, data locality is never reachable. > And there are some configurations in Spark scheduler to reduce waiting time > for data locality(e.g. "spark.locality.wait"). While, problem is that, in > listing file phase, the location informations of all the files, with all the > blocks inside each file, are all fetched from the distributed file system. > Actually, in a PROD environment, a table can be so huge that even fetching > all these location informations need take tens of seconds. > To improve such scenario, Spark need provide an option, where data locality > can be totally ignored, all we need in the listing file phase are the files > locations, without any block location informations. > > And we made a benchmark in our PROD env, after ignore the block locations, we > got a pretty huge improvement. > ||Table Size||Total File Number||Total Block Number||List File Duration(With > Block Location)||List File Duration(Without Block Location)|| > |22.6T|3|12|16.841s|1.730s| > |28.8 T|42001|148964|10.099s|2.858s| > |3.4 T|2| 2|5.833s|4.881s| > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-29189) Add an option to ignore block locations when listing file
[ https://issues.apache.org/jira/browse/SPARK-29189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-29189. -- Fix Version/s: 3.0.0 Resolution: Fixed > Add an option to ignore block locations when listing file > - > > Key: SPARK-29189 > URL: https://issues.apache.org/jira/browse/SPARK-29189 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.0.0 >Reporter: Wang, Gang >Assignee: Wang, Gang >Priority: Major > Fix For: 3.0.0 > > > In our PROD env, we have a pure Spark cluster, I think this is also pretty > common, where computation is separated from storage layer. In such deploy > mode, data locality is never reachable. > And there are some configurations in Spark scheduler to reduce waiting time > for data locality(e.g. "spark.locality.wait"). While, problem is that, in > listing file phase, the location informations of all the files, with all the > blocks inside each file, are all fetched from the distributed file system. > Actually, in a PROD environment, a table can be so huge that even fetching > all these location informations need take tens of seconds. > To improve such scenario, Spark need provide an option, where data locality > can be totally ignored, all we need in the listing file phase are the files > locations, without any block location informations. > > And we made a benchmark in our PROD env, after ignore the block locations, we > got a pretty huge improvement. > ||Table Size||Total File Number||Total Block Number||List File Duration(With > Block Location)||List File Duration(Without Block Location)|| > |22.6T|3|12|16.841s|1.730s| > |28.8 T|42001|148964|10.099s|2.858s| > |3.4 T|2| 2|5.833s|4.881s| > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-28917) Jobs can hang because of race of RDD.dependencies
[
https://issues.apache.org/jira/browse/SPARK-28917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Imran Rashid updated SPARK-28917:
-
Description:
{{RDD.dependencies}} stores the precomputed cache value, but it is not
thread-safe. This can lead to a race where the value gets overwritten, but the
DAGScheduler gets stuck in an inconsistent state. In particular, this can
happen when there is a race between the DAGScheduler event loop, and another
thread (eg. a user thread, if there is multi-threaded job submission).
First, a job is submitted by the user, which then computes the result Stage and
its parents:
https://github.com/apache/spark/blob/24655583f1cb5dae2e80bb572604fb4a9761ec07/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L983
Which eventually makes a call to {{rdd.dependencies}}:
https://github.com/apache/spark/blob/24655583f1cb5dae2e80bb572604fb4a9761ec07/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L519
At the same time, the user could also touch {{rdd.dependencies}} in another
thread, which could overwrite the stored value because of the race.
Then the DAGScheduler checks the dependencies *again* later on in the job
submission, via {{getMissingParentStages}}
https://github.com/apache/spark/blob/24655583f1cb5dae2e80bb572604fb4a9761ec07/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L1025
Because it will find new dependencies, it will create entirely different
stages. Now the job has some orphaned stages which will never run.
One symptoms of this are seeing disjoint sets of stages in the "Parents of
final stage" and the "Missing parents" messages on job submission (however this
is not required).
(*EDIT*: Seeing repeated msgs "Registering RDD X" actually is just fine, it is
not a symptom of a problem at all. It just means the RDD is the *input* to
multiple shuffles.)
{noformat}
[INFO] 2019-08-15 23:22:31,570 org.apache.spark.SparkContext logInfo - Starting
job: count at XXX.scala:462
...
[INFO] 2019-08-15 23:22:31,573 org.apache.spark.scheduler.DAGScheduler logInfo
- Registering RDD 14 (repartition at XXX.scala:421)
...
...
[INFO] 2019-08-15 23:22:31,582 org.apache.spark.scheduler.DAGScheduler logInfo
- Got job 1 (count at XXX.scala:462) with 40 output partitions
[INFO] 2019-08-15 23:22:31,582 org.apache.spark.scheduler.DAGScheduler logInfo
- Final stage: ResultStage 5 (count at XXX.scala:462)
[INFO] 2019-08-15 23:22:31,582 org.apache.spark.scheduler.DAGScheduler logInfo
- Parents of final stage: List(ShuffleMapStage 4)
[INFO] 2019-08-15 23:22:31,599 org.apache.spark.scheduler.DAGScheduler logInfo
- Registering RDD 14 (repartition at XXX.scala:421)
[INFO] 2019-08-15 23:22:31,599 org.apache.spark.scheduler.DAGScheduler logInfo
- Missing parents: List(ShuffleMapStage 6)
{noformat}
Another symptom is only visible with DEBUG logs turned on for DAGScheduler --
you will calls to {{submitStage(Stage X)}} multiple times, followed by a
different set of missing stages. eg. here, we see stage 1 first is missing
stage 0 as a dependency, and then later on its missing stage 23:
{noformat}
19/09/19 22:28:15 DEBUG scheduler.DAGScheduler: submitStage(ShuffleMapStage 1)
19/09/19 22:28:15 DEBUG scheduler.DAGScheduler: missing: List(ShuffleMapStage 0)
...
19/09/19 22:32:01 DEBUG scheduler.DAGScheduler: submitStage(ShuffleMapStage 1)
19/09/19 22:32:01 DEBUG scheduler.DAGScheduler: missing: List(ShuffleMapStage
23)
{noformat}
Note that there is a similar issue w/ {{rdd.partitions}}. In particular for
some RDDs, {{partitions}} references {{dependencies}} (eg. {{CoGroupedRDD}}).
There is also an issue that {{rdd.storageLevel}} is read and cached in the
scheduler, but it could be modified simultaneously by the user in another
thread. But, I can't see a way it could effect the scheduler.
*WORKAROUND*:
(a) call {{rdd.dependencies}} while you know that RDD is only getting touched
by one thread (eg. in the thread that created it, or before you submit multiple
jobs touching that RDD from other threads). Then that value will get cached.
(b) don't submit jobs from multiple threads.
was:
{{RDD.dependencies}} stores the precomputed cache value, but it is not
thread-safe. This can lead to a race where the value gets overwritten, but the
DAGScheduler gets stuck in an inconsistent state. In particular, this can
happen when there is a race between the DAGScheduler event loop, and another
thread (eg. a user thread, if there is multi-threaded job submission).
First, a job is submitted by the user, which then computes the result Stage and
its parents:
https://github.com/apache/spark/blob/24655583f1cb5dae2e80bb572604fb4a9761ec07/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L983
Which eventually makes a call to {{rdd.dependencies}}:
[jira] [Updated] (SPARK-28917) Jobs can hang because of race of RDD.dependencies
[
https://issues.apache.org/jira/browse/SPARK-28917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Imran Rashid updated SPARK-28917:
-
Description:
{{RDD.dependencies}} stores the precomputed cache value, but it is not
thread-safe. This can lead to a race where the value gets overwritten, but the
DAGScheduler gets stuck in an inconsistent state. In particular, this can
happen when there is a race between the DAGScheduler event loop, and another
thread (eg. a user thread, if there is multi-threaded job submission).
First, a job is submitted by the user, which then computes the result Stage and
its parents:
https://github.com/apache/spark/blob/24655583f1cb5dae2e80bb572604fb4a9761ec07/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L983
Which eventually makes a call to {{rdd.dependencies}}:
https://github.com/apache/spark/blob/24655583f1cb5dae2e80bb572604fb4a9761ec07/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L519
At the same time, the user could also touch {{rdd.dependencies}} in another
thread, which could overwrite the stored value because of the race.
Then the DAGScheduler checks the dependencies *again* later on in the job
submission, via {{getMissingParentStages}}
https://github.com/apache/spark/blob/24655583f1cb5dae2e80bb572604fb4a9761ec07/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L1025
Because it will find new dependencies, it will create entirely different
stages. Now the job has some orphaned stages which will never run.
The symptoms of this are seeing disjoint sets of stages in the "Parents of
final stage" and the "Missing parents" messages on job submission.
(*EDIT*: Seeing repeated msgs "Registering RDD X" actually is just fine, it is
not a symptom of a problem at all. It just means the RDD is the *input* to
multiple shuffles.)
{noformat}
[INFO] 2019-08-15 23:22:31,570 org.apache.spark.SparkContext logInfo - Starting
job: count at XXX.scala:462
...
[INFO] 2019-08-15 23:22:31,573 org.apache.spark.scheduler.DAGScheduler logInfo
- Registering RDD 14 (repartition at XXX.scala:421)
...
...
[INFO] 2019-08-15 23:22:31,582 org.apache.spark.scheduler.DAGScheduler logInfo
- Got job 1 (count at XXX.scala:462) with 40 output partitions
[INFO] 2019-08-15 23:22:31,582 org.apache.spark.scheduler.DAGScheduler logInfo
- Final stage: ResultStage 5 (count at XXX.scala:462)
[INFO] 2019-08-15 23:22:31,582 org.apache.spark.scheduler.DAGScheduler logInfo
- Parents of final stage: List(ShuffleMapStage 4)
[INFO] 2019-08-15 23:22:31,599 org.apache.spark.scheduler.DAGScheduler logInfo
- Registering RDD 14 (repartition at XXX.scala:421)
[INFO] 2019-08-15 23:22:31,599 org.apache.spark.scheduler.DAGScheduler logInfo
- Missing parents: List(ShuffleMapStage 6)
{noformat}
Note that there is a similar issue w/ {{rdd.partitions}}. I don't see a way it
could mess up the scheduler (seems its only used for
{{rdd.partitions.length}}). There is also an issue that {{rdd.storageLevel}}
is read and cached in the scheduler, but it could be modified simultaneously by
the user in another thread. Similarly, I can't see a way it could effect the
scheduler.
*WORKAROUND*:
(a) call {{rdd.dependencies}} while you know that RDD is only getting touched
by one thread (eg. in the thread that created it, or before you submit multiple
jobs touching that RDD from other threads). Then that value will get cached.
(b) don't submit jobs from multiple threads.
was:
{{RDD.dependencies}} stores the precomputed cache value, but it is not
thread-safe. This can lead to a race where the value gets overwritten, but the
DAGScheduler gets stuck in an inconsistent state. In particular, this can
happen when there is a race between the DAGScheduler event loop, and another
thread (eg. a user thread, if there is multi-threaded job submission).
First, a job is submitted by the user, which then computes the result Stage and
its parents:
https://github.com/apache/spark/blob/24655583f1cb5dae2e80bb572604fb4a9761ec07/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L983
Which eventually makes a call to {{rdd.dependencies}}:
https://github.com/apache/spark/blob/24655583f1cb5dae2e80bb572604fb4a9761ec07/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L519
At the same time, the user could also touch {{rdd.dependencies}} in another
thread, which could overwrite the stored value because of the race.
Then the DAGScheduler checks the dependencies *again* later on in the job
submission, via {{getMissingParentStages}}
https://github.com/apache/spark/blob/24655583f1cb5dae2e80bb572604fb4a9761ec07/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L1025
Because it will find new dependencies, it will create entirely different
stages. Now the job has some orphaned stages which will never run.
The symptoms of this are seeing disjoint sets of
[jira] [Commented] (SPARK-28592) Mark new Shuffle apis as @Experimental (instead of @Private)
[
https://issues.apache.org/jira/browse/SPARK-28592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16938028#comment-16938028
]
Imran Rashid commented on SPARK-28592:
--
[~holden] no, I think this is intentionally *not* a blocker. The idea is that
we can merge changes from the new api, even if the entire thing doesn't make it
into 3.0. We specifically do not want to block 3.0 on getting the entire thing
in (it would be great if we made it in, but don't think it should hold up the
release).
> Mark new Shuffle apis as @Experimental (instead of @Private)
>
>
> Key: SPARK-28592
> URL: https://issues.apache.org/jira/browse/SPARK-28592
> Project: Spark
> Issue Type: Sub-task
> Components: Shuffle, Spark Core
>Affects Versions: 3.0.0
>Reporter: Imran Rashid
>Priority: Major
>
> The new Shuffle api is initially marked as {{@Private}}, just to discourage
> anyone from trying to make use of it before all the pieces are in place (in
> particular if spark 3.0 is released before everything is merged). But once
> its all merged we can change to {{@Experimental}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-28917) Jobs can hang because of race of RDD.dependencies
[
https://issues.apache.org/jira/browse/SPARK-28917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930616#comment-16930616
]
Imran Rashid commented on SPARK-28917:
--
I finally got some more info about this case -- they are not using
checkpointing, nor touching dependencies. It seems things work consistently
once they move the call to {{rdd.cache()}} before touching the RDD with
multiple threads, so it could still be that caching alone is enough to mess
this up somehow.
Just by coincidence, another entirely separate group is reporting what looks to
be a very similar bug with submitting jobs from multiple threads. Its not
exactly the same, though -- it doesn't have the orphaned stages in the logs,
but does have repeated RDD registration.
> Jobs can hang because of race of RDD.dependencies
> -
>
> Key: SPARK-28917
> URL: https://issues.apache.org/jira/browse/SPARK-28917
> Project: Spark
> Issue Type: Bug
> Components: Scheduler
>Affects Versions: 2.3.3, 2.4.3
>Reporter: Imran Rashid
>Priority: Major
>
> {{RDD.dependencies}} stores the precomputed cache value, but it is not
> thread-safe. This can lead to a race where the value gets overwritten, but
> the DAGScheduler gets stuck in an inconsistent state. In particular, this
> can happen when there is a race between the DAGScheduler event loop, and
> another thread (eg. a user thread, if there is multi-threaded job submission).
> First, a job is submitted by the user, which then computes the result Stage
> and its parents:
> https://github.com/apache/spark/blob/24655583f1cb5dae2e80bb572604fb4a9761ec07/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L983
> Which eventually makes a call to {{rdd.dependencies}}:
> https://github.com/apache/spark/blob/24655583f1cb5dae2e80bb572604fb4a9761ec07/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L519
> At the same time, the user could also touch {{rdd.dependencies}} in another
> thread, which could overwrite the stored value because of the race.
> Then the DAGScheduler checks the dependencies *again* later on in the job
> submission, via {{getMissingParentStages}}
> https://github.com/apache/spark/blob/24655583f1cb5dae2e80bb572604fb4a9761ec07/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L1025
> Because it will find new dependencies, it will create entirely different
> stages. Now the job has some orphaned stages which will never run.
> The symptoms of this are seeing disjoint sets of stages in the "Parents of
> final stage" and the "Missing parents" messages on job submission, as well as
> seeing repeated messages "Registered RDD X" for the same RDD id. eg:
> {noformat}
> [INFO] 2019-08-15 23:22:31,570 org.apache.spark.SparkContext logInfo -
> Starting job: count at XXX.scala:462
> ...
> [INFO] 2019-08-15 23:22:31,573 org.apache.spark.scheduler.DAGScheduler
> logInfo - Registering RDD 14 (repartition at XXX.scala:421)
> ...
> ...
> [INFO] 2019-08-15 23:22:31,582 org.apache.spark.scheduler.DAGScheduler
> logInfo - Got job 1 (count at XXX.scala:462) with 40 output partitions
> [INFO] 2019-08-15 23:22:31,582 org.apache.spark.scheduler.DAGScheduler
> logInfo - Final stage: ResultStage 5 (count at XXX.scala:462)
> [INFO] 2019-08-15 23:22:31,582 org.apache.spark.scheduler.DAGScheduler
> logInfo - Parents of final stage: List(ShuffleMapStage 4)
> [INFO] 2019-08-15 23:22:31,599 org.apache.spark.scheduler.DAGScheduler
> logInfo - Registering RDD 14 (repartition at XXX.scala:421)
> [INFO] 2019-08-15 23:22:31,599 org.apache.spark.scheduler.DAGScheduler
> logInfo - Missing parents: List(ShuffleMapStage 6)
> {noformat}
> Note that there is a similar issue w/ {{rdd.partitions}}. I don't see a way
> it could mess up the scheduler (seems its only used for
> {{rdd.partitions.length}}). There is also an issue that {{rdd.storageLevel}}
> is read and cached in the scheduler, but it could be modified simultaneously
> by the user in another thread. Similarly, I can't see a way it could effect
> the scheduler.
> *WORKAROUND*:
> (a) call {{rdd.dependencies}} while you know that RDD is only getting touched
> by one thread (eg. in the thread that created it, or before you submit
> multiple jobs touching that RDD from other threads). Then that value will get
> cached.
> (b) don't submit jobs from multiple threads.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-29072) Properly track shuffle write time with refactor
[ https://issues.apache.org/jira/browse/SPARK-29072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-29072. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25780 [https://github.com/apache/spark/pull/25780] > Properly track shuffle write time with refactor > --- > > Key: SPARK-29072 > URL: https://issues.apache.org/jira/browse/SPARK-29072 > Project: Spark > Issue Type: Bug > Components: Shuffle >Affects Versions: 3.0.0 >Reporter: Matt Cheah >Assignee: Matt Cheah >Priority: Major > Fix For: 3.0.0 > > > From SPARK-28209, SPARK-28570, and SPARK-28571, we used the new shuffle > writer plugin API across all the shuffle writers. However, we accidentally > lost time tracking metrics for shuffle writes in the process, particularly > for UnsafeShuffleWriter when writing with streams (without transferTo), as > well as the SortShuffleWriter. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-29072) Properly track shuffle write time with refactor
[ https://issues.apache.org/jira/browse/SPARK-29072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reassigned SPARK-29072: Assignee: Matt Cheah > Properly track shuffle write time with refactor > --- > > Key: SPARK-29072 > URL: https://issues.apache.org/jira/browse/SPARK-29072 > Project: Spark > Issue Type: Bug > Components: Shuffle >Affects Versions: 3.0.0 >Reporter: Matt Cheah >Assignee: Matt Cheah >Priority: Major > > From SPARK-28209, SPARK-28570, and SPARK-28571, we used the new shuffle > writer plugin API across all the shuffle writers. However, we accidentally > lost time tracking metrics for shuffle writes in the process, particularly > for UnsafeShuffleWriter when writing with streams (without transferTo), as > well as the SortShuffleWriter. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-29017) JobGroup and LocalProperty not respected by PySpark
[
https://issues.apache.org/jira/browse/SPARK-29017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Imran Rashid resolved SPARK-29017.
--
Resolution: Duplicate
> JobGroup and LocalProperty not respected by PySpark
> ---
>
> Key: SPARK-29017
> URL: https://issues.apache.org/jira/browse/SPARK-29017
> Project: Spark
> Issue Type: Bug
> Components: PySpark
>Affects Versions: 2.4.4
>Reporter: Imran Rashid
>Priority: Major
>
> Pyspark has {{setJobGroup}} and {{setLocalProperty}} methods, which are
> intended to set properties which only effect the calling thread. They try to
> do this my calling the equivalent JVM functions via Py4J.
> However, there is nothing ensuring that subsequent py4j calls from a python
> thread call into the same thread in java. In effect, this means this methods
> might appear to work some of the time, if you happen to get lucky and get the
> same thread on the java side. But then sometimes it won't work, and in fact
> its less likely to work if there are multiple threads in python submitting
> jobs.
> I think the right way to fix this is to keep a *python* thread-local tracking
> these properties, and then sending them through to the JVM on calls to
> submitJob. This is going to be a headache to get right, though; we've also
> got to handle implicit calls, eg. {{rdd.collect()}}, {{rdd.forEach()}}, etc.
> And of course users may have defined their own functions, which will be
> broken until they fix it to use the same thread-locals.
> An alternative might be to use what py4j calls the "Single Threading Model"
> (https://www.py4j.org/advanced_topics.html#the-single-threading-model). I'd
> want to look more closely at the py4j implementation of how that works first.
> I can't think of any guaranteed workaround, but I think you could increase
> your chances of getting the desired behavior if you always set those
> properties just before each call to runJob. Eg., instead of
> {code:python}
> # more likely to trigger bug this way
> sc.setJobGroup("a")
> rdd1.collect() # or whatever other ways you submit a job
> rdd2.collect()
> # lots more stuff ...
> rddN.collect()
> {code}
> change it to
> {code:python}
> # slightly safer, but still no guarantees
> sc.setJobGroup("a")
> rdd1.collect() # or whatever other ways you submit a job
> sc.setJobGroup("a")
> rdd2.collect()
> # lots more stuff ...
> sc.setJobGroup("a")
> rddN.collect()
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-29017) JobGroup and LocalProperty not respected by PySpark
[
https://issues.apache.org/jira/browse/SPARK-29017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16924975#comment-16924975
]
Imran Rashid commented on SPARK-29017:
--
[~hyukjin.kwon][~holdensmagicalunicorn] [~bryanc] any thoughts on how to fix
this issue in py4j?
[~felixcheung] I don't think this issue exists in SparkR, because I didn't see
anything like {{setJobGroup}} or {{setLocalProperty}} but maybe I'm not looking
in the right place.
> JobGroup and LocalProperty not respected by PySpark
> ---
>
> Key: SPARK-29017
> URL: https://issues.apache.org/jira/browse/SPARK-29017
> Project: Spark
> Issue Type: Bug
> Components: PySpark
>Affects Versions: 2.4.4
>Reporter: Imran Rashid
>Priority: Major
>
> Pyspark has {{setJobGroup}} and {{setLocalProperty}} methods, which are
> intended to set properties which only effect the calling thread. They try to
> do this my calling the equivalent JVM functions via Py4J.
> However, there is nothing ensuring that subsequent py4j calls from a python
> thread call into the same thread in java. In effect, this means this methods
> might appear to work some of the time, if you happen to get lucky and get the
> same thread on the java side. But then sometimes it won't work, and in fact
> its less likely to work if there are multiple threads in python submitting
> jobs.
> I think the right way to fix this is to keep a *python* thread-local tracking
> these properties, and then sending them through to the JVM on calls to
> submitJob. This is going to be a headache to get right, though; we've also
> got to handle implicit calls, eg. {{rdd.collect()}}, {{rdd.forEach()}}, etc.
> And of course users may have defined their own functions, which will be
> broken until they fix it to use the same thread-locals.
> An alternative might be to use what py4j calls the "Single Threading Model"
> (https://www.py4j.org/advanced_topics.html#the-single-threading-model). I'd
> want to look more closely at the py4j implementation of how that works first.
> I can't think of any guaranteed workaround, but I think you could increase
> your chances of getting the desired behavior if you always set those
> properties just before each call to runJob. Eg., instead of
> {code:python}
> # more likely to trigger bug this way
> sc.setJobGroup("a")
> rdd1.collect() # or whatever other ways you submit a job
> rdd2.collect()
> # lots more stuff ...
> rddN.collect()
> {code}
> change it to
> {code:python}
> # slightly safer, but still no guarantees
> sc.setJobGroup("a")
> rdd1.collect() # or whatever other ways you submit a job
> sc.setJobGroup("a")
> rdd2.collect()
> # lots more stuff ...
> sc.setJobGroup("a")
> rddN.collect()
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-29017) JobGroup and LocalProperty not respected by PySpark
[
https://issues.apache.org/jira/browse/SPARK-29017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Imran Rashid updated SPARK-29017:
-
Description:
Pyspark has {{setJobGroup}} and {{setLocalProperty}} methods, which are
intended to set properties which only effect the calling thread. They try to
do this my calling the equivalent JVM functions via Py4J.
However, there is nothing ensuring that subsequent py4j calls from a python
thread call into the same thread in java. In effect, this means this methods
might appear to work some of the time, if you happen to get lucky and get the
same thread on the java side. But then sometimes it won't work, and in fact
its less likely to work if there are multiple threads in python submitting jobs.
I think the right way to fix this is to keep a *python* thread-local tracking
these properties, and then sending them through to the JVM on calls to
submitJob. This is going to be a headache to get right, though; we've also got
to handle implicit calls, eg. {{rdd.collect()}}, {{rdd.forEach()}}, etc. And
of course users may have defined their own functions, which will be broken
until they fix it to use the same thread-locals.
An alternative might be to use what py4j calls the "Single Threading Model"
(https://www.py4j.org/advanced_topics.html#the-single-threading-model). I'd
want to look more closely at the py4j implementation of how that works first.
I can't think of any guaranteed workaround, but I think you could increase your
chances of getting the desired behavior if you always set those properties just
before each call to runJob. Eg., instead of
{code:python}
# more likely to trigger bug this way
sc.setJobGroup("a")
rdd1.collect() # or whatever other ways you submit a job
rdd2.collect()
# lots more stuff ...
rddN.collect()
{code}
change it to
{code:python}
# slightly safer, but still no guarantees
sc.setJobGroup("a")
rdd1.collect() # or whatever other ways you submit a job
sc.setJobGroup("a")
rdd2.collect()
# lots more stuff ...
sc.setJobGroup("a")
rddN.collect()
{code}
was:
Pyspark has {{setJobGroup}} and {{setLocalProperty}} methods, which are
intended to set properties which only effect the calling thread. They try to
do this my calling the equivalent JVM functions via Py4J.
However, there is nothing ensuring that subsequent py4j calls from a python
thread call into the same thread in java. In effect, this means this methods
might appear to work some of the time, if you happen to get lucky and get the
same thread on the java side. But then sometimes it won't work, and in fact
its less likely to work if there are multiple threads in python submitting jobs.
I think the right way to fix this is to keep a *python* thread-local tracking
these properties, and then sending them through to the JVM on calls to
submitJob. This is going to be a headache to get right, though; we've also got
to handle implicit calls, eg. {{rdd.collect()}}, {{rdd.forEach()}}, etc. And
of course users may have defined their own functions, which will be broken
until they fix it to use the same thread-locals.
An alternative might be to use what py4j calls the "Single Threading Model"
(https://www.py4j.org/advanced_topics.html#the-single-threading-model). I'd
want to look more closely at the py4j implementation of how that works first.
> JobGroup and LocalProperty not respected by PySpark
> ---
>
> Key: SPARK-29017
> URL: https://issues.apache.org/jira/browse/SPARK-29017
> Project: Spark
> Issue Type: Bug
> Components: PySpark
>Affects Versions: 2.4.4
>Reporter: Imran Rashid
>Priority: Major
>
> Pyspark has {{setJobGroup}} and {{setLocalProperty}} methods, which are
> intended to set properties which only effect the calling thread. They try to
> do this my calling the equivalent JVM functions via Py4J.
> However, there is nothing ensuring that subsequent py4j calls from a python
> thread call into the same thread in java. In effect, this means this methods
> might appear to work some of the time, if you happen to get lucky and get the
> same thread on the java side. But then sometimes it won't work, and in fact
> its less likely to work if there are multiple threads in python submitting
> jobs.
> I think the right way to fix this is to keep a *python* thread-local tracking
> these properties, and then sending them through to the JVM on calls to
> submitJob. This is going to be a headache to get right, though; we've also
> got to handle implicit calls, eg. {{rdd.collect()}}, {{rdd.forEach()}}, etc.
> And of course users may have defined their own functions, which will be
> broken until they fix it to use the same thread-locals.
> An alternative might be to use what py4j calls the "Single Threading Model"
>
[jira] [Created] (SPARK-29017) JobGroup and LocalProperty not respected by PySpark
Imran Rashid created SPARK-29017:
Summary: JobGroup and LocalProperty not respected by PySpark
Key: SPARK-29017
URL: https://issues.apache.org/jira/browse/SPARK-29017
Project: Spark
Issue Type: Bug
Components: PySpark
Affects Versions: 2.4.4
Reporter: Imran Rashid
Pyspark has {{setJobGroup}} and {{setLocalProperty}} methods, which are
intended to set properties which only effect the calling thread. They try to
do this my calling the equivalent JVM functions via Py4J.
However, there is nothing ensuring that subsequent py4j calls from a python
thread call into the same thread in java. In effect, this means this methods
might appear to work some of the time, if you happen to get lucky and get the
same thread on the java side. But then sometimes it won't work, and in fact
its less likely to work if there are multiple threads in python submitting jobs.
I think the right way to fix this is to keep a *python* thread-local tracking
these properties, and then sending them through to the JVM on calls to
submitJob. This is going to be a headache to get right, though; we've also got
to handle implicit calls, eg. {{rdd.collect()}}, {{rdd.forEach()}}, etc. And
of course users may have defined their own functions, which will be broken
until they fix it to use the same thread-locals.
An alternative might be to use what py4j calls the "Single Threading Model"
(https://www.py4j.org/advanced_topics.html#the-single-threading-model). I'd
want to look more closely at the py4j implementation of how that works first.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-28770) Flaky Tests: Test ReplayListenerSuite.End-to-end replay with compression failed
[ https://issues.apache.org/jira/browse/SPARK-28770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-28770. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25673 [https://github.com/apache/spark/pull/25673] > Flaky Tests: Test ReplayListenerSuite.End-to-end replay with compression > failed > --- > > Key: SPARK-28770 > URL: https://issues.apache.org/jira/browse/SPARK-28770 > Project: Spark > Issue Type: Test > Components: Spark Core >Affects Versions: 2.4.3 > Environment: Community jenkins and our arm testing instance. >Reporter: huangtianhua >Assignee: Wing Yew Poon >Priority: Major > Fix For: 3.0.0 > > > Test > org.apache.spark.scheduler.ReplayListenerSuite.End-to-end replay with > compression is failed see > [https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-3.2/267/testReport/junit/org.apache.spark.scheduler/ReplayListenerSuite/End_to_end_replay_with_compression/] > > And also the test is failed on arm instance, I sent email to spark-dev > before, and we suspect there is something related with the commit > [https://github.com/apache/spark/pull/23767], we tried to revert it and the > tests are passed: > ReplayListenerSuite: > - ... > - End-to-end replay *** FAILED *** > "[driver]" did not equal "[1]" (JsonProtocolSuite.scala:622) > - End-to-end replay with compression *** FAILED *** > "[driver]" did not equal "[1]" (JsonProtocolSuite.scala:622) > > Not sure what's wrong, hope someone can help to figure it out, thanks very > much. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-28770) Flaky Tests: Test ReplayListenerSuite.End-to-end replay with compression failed
[ https://issues.apache.org/jira/browse/SPARK-28770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reassigned SPARK-28770: Assignee: Wing Yew Poon > Flaky Tests: Test ReplayListenerSuite.End-to-end replay with compression > failed > --- > > Key: SPARK-28770 > URL: https://issues.apache.org/jira/browse/SPARK-28770 > Project: Spark > Issue Type: Test > Components: Spark Core >Affects Versions: 2.4.3 > Environment: Community jenkins and our arm testing instance. >Reporter: huangtianhua >Assignee: Wing Yew Poon >Priority: Major > > Test > org.apache.spark.scheduler.ReplayListenerSuite.End-to-end replay with > compression is failed see > [https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-3.2/267/testReport/junit/org.apache.spark.scheduler/ReplayListenerSuite/End_to_end_replay_with_compression/] > > And also the test is failed on arm instance, I sent email to spark-dev > before, and we suspect there is something related with the commit > [https://github.com/apache/spark/pull/23767], we tried to revert it and the > tests are passed: > ReplayListenerSuite: > - ... > - End-to-end replay *** FAILED *** > "[driver]" did not equal "[1]" (JsonProtocolSuite.scala:622) > - End-to-end replay with compression *** FAILED *** > "[driver]" did not equal "[1]" (JsonProtocolSuite.scala:622) > > Not sure what's wrong, hope someone can help to figure it out, thanks very > much. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-28770) Flaky Tests: Test ReplayListenerSuite.End-to-end replay with compression failed
[ https://issues.apache.org/jira/browse/SPARK-28770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16922825#comment-16922825 ] Imran Rashid commented on SPARK-28770: -- Wing yew and I chatted about this offline a bit. I follow his reasoning now. We also think [~kabhwan] has a good point. I think the connection between EventLoggingListener and EventMonster is probably not so good, it is from long ago (eg. before there was a SparkFirehoseListener). So he's gonna try to explore changing it do the right thing and update the PR (assuming no unexpected surprises) > Flaky Tests: Test ReplayListenerSuite.End-to-end replay with compression > failed > --- > > Key: SPARK-28770 > URL: https://issues.apache.org/jira/browse/SPARK-28770 > Project: Spark > Issue Type: Test > Components: Spark Core >Affects Versions: 2.4.3 > Environment: Community jenkins and our arm testing instance. >Reporter: huangtianhua >Priority: Major > > Test > org.apache.spark.scheduler.ReplayListenerSuite.End-to-end replay with > compression is failed see > [https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-3.2/267/testReport/junit/org.apache.spark.scheduler/ReplayListenerSuite/End_to_end_replay_with_compression/] > > And also the test is failed on arm instance, I sent email to spark-dev > before, and we suspect there is something related with the commit > [https://github.com/apache/spark/pull/23767], we tried to revert it and the > tests are passed: > ReplayListenerSuite: > - ... > - End-to-end replay *** FAILED *** > "[driver]" did not equal "[1]" (JsonProtocolSuite.scala:622) > - End-to-end replay with compression *** FAILED *** > "[driver]" did not equal "[1]" (JsonProtocolSuite.scala:622) > > Not sure what's wrong, hope someone can help to figure it out, thanks very > much. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-28770) Flaky Tests: Test ReplayListenerSuite.End-to-end replay with compression failed
[
https://issues.apache.org/jira/browse/SPARK-28770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16922771#comment-16922771
]
Imran Rashid commented on SPARK-28770:
--
I'm not sure I follow ... the metrics from the driver will be in the log file
if and only if {{park.eventLog.logStageExecutorMetrics.enabled=true}} and there
is a stage completed event. But so that same criteria should apply on replay,
so the EventMonster would also "log" the metrics update (meaning, record it in
its internal buffer) if and only if the event log file also contained the stage
completed event.
[~bzhaoopenstack] since you can reproduce this consistently, could you share
the event log file from a failure? I think you just need to delete this block
in ReplayListenerQuite:
{code}
after {
Utils.deleteRecursively(testDir)
}
{code}
and maybe print out that dir somewhere so you can grab the file.
> Flaky Tests: Test ReplayListenerSuite.End-to-end replay with compression
> failed
> ---
>
> Key: SPARK-28770
> URL: https://issues.apache.org/jira/browse/SPARK-28770
> Project: Spark
> Issue Type: Test
> Components: Spark Core
>Affects Versions: 2.4.3
> Environment: Community jenkins and our arm testing instance.
>Reporter: huangtianhua
>Priority: Major
>
> Test
> org.apache.spark.scheduler.ReplayListenerSuite.End-to-end replay with
> compression is failed see
> [https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-3.2/267/testReport/junit/org.apache.spark.scheduler/ReplayListenerSuite/End_to_end_replay_with_compression/]
>
> And also the test is failed on arm instance, I sent email to spark-dev
> before, and we suspect there is something related with the commit
> [https://github.com/apache/spark/pull/23767], we tried to revert it and the
> tests are passed:
> ReplayListenerSuite:
> - ...
> - End-to-end replay *** FAILED ***
> "[driver]" did not equal "[1]" (JsonProtocolSuite.scala:622)
> - End-to-end replay with compression *** FAILED ***
> "[driver]" did not equal "[1]" (JsonProtocolSuite.scala:622)
>
> Not sure what's wrong, hope someone can help to figure it out, thanks very
> much.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-28917) Jobs can hang because of race of RDD.dependencies
Imran Rashid created SPARK-28917:
Summary: Jobs can hang because of race of RDD.dependencies
Key: SPARK-28917
URL: https://issues.apache.org/jira/browse/SPARK-28917
Project: Spark
Issue Type: Bug
Components: Scheduler
Affects Versions: 2.4.3, 2.3.3
Reporter: Imran Rashid
{{RDD.dependencies}} stores the precomputed cache value, but it is not
thread-safe. This can lead to a race where the value gets overwritten, but the
DAGScheduler gets stuck in an inconsistent state. In particular, this can
happen when there is a race between the DAGScheduler event loop, and another
thread (eg. a user thread, if there is multi-threaded job submission).
First, a job is submitted by the user, which then computes the result Stage and
its parents:
https://github.com/apache/spark/blob/24655583f1cb5dae2e80bb572604fb4a9761ec07/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L983
Which eventually makes a call to {{rdd.dependencies}}:
https://github.com/apache/spark/blob/24655583f1cb5dae2e80bb572604fb4a9761ec07/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L519
At the same time, the user could also touch {{rdd.dependencies}} in another
thread, which could overwrite the stored value because of the race.
Then the DAGScheduler checks the dependencies *again* later on in the job
submission, via {{getMissingParentStages}}
https://github.com/apache/spark/blob/24655583f1cb5dae2e80bb572604fb4a9761ec07/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L1025
Because it will find new dependencies, it will create entirely different
stages. Now the job has some orphaned stages which will never run.
The symptoms of this are seeing disjoint sets of stages in the "Parents of
final stage" and the "Missing parents" messages on job submission, as well as
seeing repeated messages "Registered RDD X" for the same RDD id. eg:
{noformat}
[INFO] 2019-08-15 23:22:31,570 org.apache.spark.SparkContext logInfo - Starting
job: count at XXX.scala:462
...
[INFO] 2019-08-15 23:22:31,573 org.apache.spark.scheduler.DAGScheduler logInfo
- Registering RDD 14 (repartition at XXX.scala:421)
...
...
[INFO] 2019-08-15 23:22:31,582 org.apache.spark.scheduler.DAGScheduler logInfo
- Got job 1 (count at XXX.scala:462) with 40 output partitions
[INFO] 2019-08-15 23:22:31,582 org.apache.spark.scheduler.DAGScheduler logInfo
- Final stage: ResultStage 5 (count at XXX.scala:462)
[INFO] 2019-08-15 23:22:31,582 org.apache.spark.scheduler.DAGScheduler logInfo
- Parents of final stage: List(ShuffleMapStage 4)
[INFO] 2019-08-15 23:22:31,599 org.apache.spark.scheduler.DAGScheduler logInfo
- Registering RDD 14 (repartition at XXX.scala:421)
[INFO] 2019-08-15 23:22:31,599 org.apache.spark.scheduler.DAGScheduler logInfo
- Missing parents: List(ShuffleMapStage 6)
{noformat}
Note that there is a similar issue w/ {{rdd.partitions}}. I don't see a way it
could mess up the scheduler (seems its only used for
{{rdd.partitions.length}}). There is also an issue that {{rdd.storageLevel}}
is read and cached in the scheduler, but it could be modified simultaneously by
the user in another thread. Similarly, I can't see a way it could effect the
scheduler.
*WORKAROUND*:
(a) call {{rdd.dependencies}} while you know that RDD is only getting touched
by one thread (eg. in the thread that created it, or before you submit multiple
jobs touching that RDD from other threads). Then that value will get cached.
(b) don't submit jobs from multiple threads.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-28917) Jobs can hang because of race of RDD.dependencies
[
https://issues.apache.org/jira/browse/SPARK-28917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918764#comment-16918764
]
Imran Rashid commented on SPARK-28917:
--
[~markhamstra] [~jiangxb1987] [~tgraves] [~Ngone51] would appreciate your
thoughts on this. I think the bug I've described above is pretty clear.
However, the part which I'm wondering about a bit more is whether there is more
mutability in RDD that could cause problems.
For the case I have of this, I only know for sure that the user is calling
{{rdd.cache()}} in another thread. But I can't see how that would leave to the
symptoms I describe above. I don't know that they are doing anything in ther
user thread which would touch {{rdd.dependencies}}, but I also don't have full
visibility into everything they are doing, so this still seems like the best
explanation to me.
> Jobs can hang because of race of RDD.dependencies
> -
>
> Key: SPARK-28917
> URL: https://issues.apache.org/jira/browse/SPARK-28917
> Project: Spark
> Issue Type: Bug
> Components: Scheduler
>Affects Versions: 2.3.3, 2.4.3
>Reporter: Imran Rashid
>Priority: Major
>
> {{RDD.dependencies}} stores the precomputed cache value, but it is not
> thread-safe. This can lead to a race where the value gets overwritten, but
> the DAGScheduler gets stuck in an inconsistent state. In particular, this
> can happen when there is a race between the DAGScheduler event loop, and
> another thread (eg. a user thread, if there is multi-threaded job submission).
> First, a job is submitted by the user, which then computes the result Stage
> and its parents:
> https://github.com/apache/spark/blob/24655583f1cb5dae2e80bb572604fb4a9761ec07/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L983
> Which eventually makes a call to {{rdd.dependencies}}:
> https://github.com/apache/spark/blob/24655583f1cb5dae2e80bb572604fb4a9761ec07/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L519
> At the same time, the user could also touch {{rdd.dependencies}} in another
> thread, which could overwrite the stored value because of the race.
> Then the DAGScheduler checks the dependencies *again* later on in the job
> submission, via {{getMissingParentStages}}
> https://github.com/apache/spark/blob/24655583f1cb5dae2e80bb572604fb4a9761ec07/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L1025
> Because it will find new dependencies, it will create entirely different
> stages. Now the job has some orphaned stages which will never run.
> The symptoms of this are seeing disjoint sets of stages in the "Parents of
> final stage" and the "Missing parents" messages on job submission, as well as
> seeing repeated messages "Registered RDD X" for the same RDD id. eg:
> {noformat}
> [INFO] 2019-08-15 23:22:31,570 org.apache.spark.SparkContext logInfo -
> Starting job: count at XXX.scala:462
> ...
> [INFO] 2019-08-15 23:22:31,573 org.apache.spark.scheduler.DAGScheduler
> logInfo - Registering RDD 14 (repartition at XXX.scala:421)
> ...
> ...
> [INFO] 2019-08-15 23:22:31,582 org.apache.spark.scheduler.DAGScheduler
> logInfo - Got job 1 (count at XXX.scala:462) with 40 output partitions
> [INFO] 2019-08-15 23:22:31,582 org.apache.spark.scheduler.DAGScheduler
> logInfo - Final stage: ResultStage 5 (count at XXX.scala:462)
> [INFO] 2019-08-15 23:22:31,582 org.apache.spark.scheduler.DAGScheduler
> logInfo - Parents of final stage: List(ShuffleMapStage 4)
> [INFO] 2019-08-15 23:22:31,599 org.apache.spark.scheduler.DAGScheduler
> logInfo - Registering RDD 14 (repartition at XXX.scala:421)
> [INFO] 2019-08-15 23:22:31,599 org.apache.spark.scheduler.DAGScheduler
> logInfo - Missing parents: List(ShuffleMapStage 6)
> {noformat}
> Note that there is a similar issue w/ {{rdd.partitions}}. I don't see a way
> it could mess up the scheduler (seems its only used for
> {{rdd.partitions.length}}). There is also an issue that {{rdd.storageLevel}}
> is read and cached in the scheduler, but it could be modified simultaneously
> by the user in another thread. Similarly, I can't see a way it could effect
> the scheduler.
> *WORKAROUND*:
> (a) call {{rdd.dependencies}} while you know that RDD is only getting touched
> by one thread (eg. in the thread that created it, or before you submit
> multiple jobs touching that RDD from other threads). Then that value will get
> cached.
> (b) don't submit jobs from multiple threads.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-28626) Spark leaves unencrypted data on local disk, even with encryption turned on (CVE-2019-10099)
[
https://issues.apache.org/jira/browse/SPARK-28626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Imran Rashid updated SPARK-28626:
-
Description:
Severity: Important
Vendor: The Apache Software Foundation
Versions affected:
All Spark 1.x, Spark 2.0.x, Spark 2.1.x, and 2.2.x versions
Spark 2.3.0 to 2.3.2
Description:
Prior to Spark 2.3.3, in certain situations Spark would write user data to
local disk unencrypted, even if spark.io.encryption.enabled=true. This
includes cached blocks that are fetched to disk (controlled by
spark.maxRemoteBlockSizeFetchToMem); in SparkR, using parallelize; in Pyspark,
using broadcast and parallelize; and use of python udfs.
Mitigation:
1.x, 2.0.x, 2.1.x, 2.2.x, 2.3.x users should upgrade to 2.3.3 or newer,
including 2.4.x
Credit:
This issue was reported by Thomas Graves of NVIDIA.
References:
[https://spark.apache.org/security.html]
The following commits were used to fix this issue, in branch-2.3 (there may be
other commits in master / branch-2.4, that are equivalent.)
{noformat}
commit 575fea120e25249716e3f680396580c5f9e26b5b
Author: Imran Rashid
Date: Wed Aug 22 16:38:28 2018 -0500
[CORE] Updates to remote cache reads
Covered by tests in DistributedSuite
commit 6d742d1bd71aa3803dce91a830b37284cb18cf70
Author: Imran Rashid
Date: Thu Sep 6 12:11:47 2018 -0500
[PYSPARK][SQL] Updates to RowQueue
Tested with updates to RowQueueSuite
commit 09dd34cb1706f2477a89174d6a1a0f17ed5b0a65
Author: Imran Rashid
Date: Mon Aug 13 21:35:34 2018 -0500
[PYSPARK] Updates to pyspark broadcast
commit 12717ba0edfa5459c9ac2085f46b1ecc0ee759aa
Author: hyukjinkwon
Date: Mon Sep 24 19:25:02 2018 +0800
[SPARKR] Match pyspark features in SparkR communication protocol
{noformat}
was:
Severity: Important
Vendor: The Apache Software Foundation
Versions affected:
All Spark 1.x, Spark 2.0.x, Spark 2.1.x, and 2.2.x versions
Spark 2.3.0 to 2.3.2
Description:
Prior to Spark 2.3.3, in certain situations Spark would write user data to
local disk unencrypted, even if spark.io.encryption.enabled=true. This
includes cached blocks that are fetched to disk (controlled by
spark.maxRemoteBlockSizeFetchToMem); in SparkR, using parallelize; in Pyspark,
using broadcast and parallelize; and use of python udfs.
Mitigation:
1.x, 2.0.x, 2.1.x, 2.2.x, 2.3.x users should upgrade to 2.3.3 or newer
Credit:
This issue was reported by Thomas Graves of NVIDIA.
References:
[https://spark.apache.org/security.html]
The following commits were used to fix this issue, in branch-2.3 (there may be
other commits in master / branch-2.4, that are equivalent.)
{noformat}
commit 575fea120e25249716e3f680396580c5f9e26b5b
Author: Imran Rashid
Date: Wed Aug 22 16:38:28 2018 -0500
[CORE] Updates to remote cache reads
Covered by tests in DistributedSuite
commit 6d742d1bd71aa3803dce91a830b37284cb18cf70
Author: Imran Rashid
Date: Thu Sep 6 12:11:47 2018 -0500
[PYSPARK][SQL] Updates to RowQueue
Tested with updates to RowQueueSuite
commit 09dd34cb1706f2477a89174d6a1a0f17ed5b0a65
Author: Imran Rashid
Date: Mon Aug 13 21:35:34 2018 -0500
[PYSPARK] Updates to pyspark broadcast
commit 12717ba0edfa5459c9ac2085f46b1ecc0ee759aa
Author: hyukjinkwon
Date: Mon Sep 24 19:25:02 2018 +0800
[SPARKR] Match pyspark features in SparkR communication protocol
{noformat}
> Spark leaves unencrypted data on local disk, even with encryption turned on
> (CVE-2019-10099)
>
>
> Key: SPARK-28626
> URL: https://issues.apache.org/jira/browse/SPARK-28626
> Project: Spark
> Issue Type: Bug
> Components: Security
>Affects Versions: 2.3.2
>Reporter: Imran Rashid
>Priority: Major
> Fix For: 2.3.3, 2.4.0
>
>
> Severity: Important
>
> Vendor: The Apache Software Foundation
>
> Versions affected:
> All Spark 1.x, Spark 2.0.x, Spark 2.1.x, and 2.2.x versions
> Spark 2.3.0 to 2.3.2
>
> Description:
> Prior to Spark 2.3.3, in certain situations Spark would write user data to
> local disk unencrypted, even if spark.io.encryption.enabled=true. This
> includes cached blocks that are fetched to disk (controlled by
> spark.maxRemoteBlockSizeFetchToMem); in SparkR, using parallelize; in
> Pyspark, using broadcast and parallelize; and use of python udfs.
>
>
> Mitigation:
> 1.x, 2.0.x, 2.1.x, 2.2.x, 2.3.x users should upgrade to 2.3.3 or newer,
> including 2.4.x
>
> Credit:
> This issue was reported by Thomas Graves of NVIDIA.
>
> References:
> [https://spark.apache.org/security.html]
>
> The following commits were used to fix this issue, in branch-2.3 (there may
> be other commits in master / branch-2.4, that are
[jira] [Updated] (SPARK-28626) Spark leaves unencrypted data on local disk, even with encryption turned on (CVE-2019-10099)
[
https://issues.apache.org/jira/browse/SPARK-28626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Imran Rashid updated SPARK-28626:
-
Description:
Severity: Important
Vendor: The Apache Software Foundation
Versions affected:
All Spark 1.x, Spark 2.0.x, Spark 2.1.x, and 2.2.x versions
Spark 2.3.0 to 2.3.2
Description:
Prior to Spark 2.3.3, in certain situations Spark would write user data to
local disk unencrypted, even if spark.io.encryption.enabled=true. This
includes cached blocks that are fetched to disk (controlled by
spark.maxRemoteBlockSizeFetchToMem); in SparkR, using parallelize; in Pyspark,
using broadcast and parallelize; and use of python udfs.
Mitigation:
1.x, 2.0.x, 2.1.x, 2.2.x, 2.3.x users should upgrade to 2.3.3 or newer
Credit:
This issue was reported by Thomas Graves of NVIDIA.
References:
[https://spark.apache.org/security.html]
The following commits were used to fix this issue, in branch-2.3 (there may be
other commits in master / branch-2.4, that are equivalent.)
{noformat}
commit 575fea120e25249716e3f680396580c5f9e26b5b
Author: Imran Rashid
Date: Wed Aug 22 16:38:28 2018 -0500
[CORE] Updates to remote cache reads
Covered by tests in DistributedSuite
commit 6d742d1bd71aa3803dce91a830b37284cb18cf70
Author: Imran Rashid
Date: Thu Sep 6 12:11:47 2018 -0500
[PYSPARK][SQL] Updates to RowQueue
Tested with updates to RowQueueSuite
commit 09dd34cb1706f2477a89174d6a1a0f17ed5b0a65
Author: Imran Rashid
Date: Mon Aug 13 21:35:34 2018 -0500
[PYSPARK] Updates to pyspark broadcast
commit 12717ba0edfa5459c9ac2085f46b1ecc0ee759aa
Author: hyukjinkwon
Date: Mon Sep 24 19:25:02 2018 +0800
[SPARKR] Match pyspark features in SparkR communication protocol
{noformat}
was:
Severity: Important
Vendor: The Apache Software Foundation
Versions affected:
All Spark 1.x, Spark 2.0.x, Spark 2.1.x, and 2.2.x versions
Spark 2.3.0 to 2.3.2
Description:
Prior to Spark 2.3.3, in certain situations Spark would write user data to
local disk unencrypted, even if spark.io.encryption.enabled=true. This
includes cached blocks that are fetched to disk (controlled by
spark.maxRemoteBlockSizeFetchToMem); in SparkR, using parallelize; in Pyspark,
using broadcast and parallelize; and use of python udfs.
Mitigation:
1.x, 2.0.x, 2.1.x, 2.2.x, 2.3.x users should upgrade to 2.3.3 or newer
2.3.x users should upgrade to 2.3.3 or newer.
Credit:
This issue was reported by Thomas Graves of NVIDIA.
References:
[https://spark.apache.org/security.html]
The following commits were used to fix this issue, in branch-2.3 (there may be
other commits in master / branch-2.4, that are equivalent.)
{noformat}
commit 575fea120e25249716e3f680396580c5f9e26b5b
Author: Imran Rashid
Date: Wed Aug 22 16:38:28 2018 -0500
[CORE] Updates to remote cache reads
Covered by tests in DistributedSuite
commit 6d742d1bd71aa3803dce91a830b37284cb18cf70
Author: Imran Rashid
Date: Thu Sep 6 12:11:47 2018 -0500
[PYSPARK][SQL] Updates to RowQueue
Tested with updates to RowQueueSuite
commit 09dd34cb1706f2477a89174d6a1a0f17ed5b0a65
Author: Imran Rashid
Date: Mon Aug 13 21:35:34 2018 -0500
[PYSPARK] Updates to pyspark broadcast
commit 12717ba0edfa5459c9ac2085f46b1ecc0ee759aa
Author: hyukjinkwon
Date: Mon Sep 24 19:25:02 2018 +0800
[SPARKR] Match pyspark features in SparkR communication protocol
{noformat}
> Spark leaves unencrypted data on local disk, even with encryption turned on
> (CVE-2019-10099)
>
>
> Key: SPARK-28626
> URL: https://issues.apache.org/jira/browse/SPARK-28626
> Project: Spark
> Issue Type: Bug
> Components: Security
>Affects Versions: 2.3.2
>Reporter: Imran Rashid
>Priority: Major
> Fix For: 2.3.3, 2.4.0
>
>
> Severity: Important
>
> Vendor: The Apache Software Foundation
>
> Versions affected:
> All Spark 1.x, Spark 2.0.x, Spark 2.1.x, and 2.2.x versions
> Spark 2.3.0 to 2.3.2
>
> Description:
> Prior to Spark 2.3.3, in certain situations Spark would write user data to
> local disk unencrypted, even if spark.io.encryption.enabled=true. This
> includes cached blocks that are fetched to disk (controlled by
> spark.maxRemoteBlockSizeFetchToMem); in SparkR, using parallelize; in
> Pyspark, using broadcast and parallelize; and use of python udfs.
>
>
> Mitigation:
> 1.x, 2.0.x, 2.1.x, 2.2.x, 2.3.x users should upgrade to 2.3.3 or newer
>
> Credit:
> This issue was reported by Thomas Graves of NVIDIA.
>
> References:
> [https://spark.apache.org/security.html]
>
> The following commits were used to fix this issue, in branch-2.3 (there may
> be other commits in master / branch-2.4, that
[jira] [Created] (SPARK-28626) Spark leaves unencrypted data on local disk, even with encryption turned on (CVE-2019-10099)
Imran Rashid created SPARK-28626:
Summary: Spark leaves unencrypted data on local disk, even with
encryption turned on (CVE-2019-10099)
Key: SPARK-28626
URL: https://issues.apache.org/jira/browse/SPARK-28626
Project: Spark
Issue Type: Bug
Components: Security
Affects Versions: 2.3.2
Reporter: Imran Rashid
Fix For: 2.4.0, 2.3.3
Severity: Important
Vendor: The Apache Software Foundation
Versions affected:
All Spark 1.x, Spark 2.0.x, Spark 2.1.x, and 2.2.x versions
Spark 2.3.0 to 2.3.2
Description:
Prior to Spark 2.3.3, in certain situations Spark would write user data to
local disk unencrypted, even if spark.io.encryption.enabled=true. This
includes cached blocks that are fetched to disk (controlled by
spark.maxRemoteBlockSizeFetchToMem); in SparkR, using parallelize; in Pyspark,
using broadcast and parallelize; and use of python udfs.
Mitigation:
1.x, 2.0.x, 2.1.x, 2.2.x, 2.3.x users should upgrade to 2.3.3 or newer
2.3.x users should upgrade to 2.3.3 or newer.
Credit:
This issue was reported by Thomas Graves of NVIDIA.
References:
[https://spark.apache.org/security.html]
The following commits were used to fix this issue, in branch-2.3 (there may be
other commits in master / branch-2.4, that are equivalent.)
{noformat}
commit 575fea120e25249716e3f680396580c5f9e26b5b
Author: Imran Rashid
Date: Wed Aug 22 16:38:28 2018 -0500
[CORE] Updates to remote cache reads
Covered by tests in DistributedSuite
commit 6d742d1bd71aa3803dce91a830b37284cb18cf70
Author: Imran Rashid
Date: Thu Sep 6 12:11:47 2018 -0500
[PYSPARK][SQL] Updates to RowQueue
Tested with updates to RowQueueSuite
commit 09dd34cb1706f2477a89174d6a1a0f17ed5b0a65
Author: Imran Rashid
Date: Mon Aug 13 21:35:34 2018 -0500
[PYSPARK] Updates to pyspark broadcast
commit 12717ba0edfa5459c9ac2085f46b1ecc0ee759aa
Author: hyukjinkwon
Date: Mon Sep 24 19:25:02 2018 +0800
[SPARKR] Match pyspark features in SparkR communication protocol
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-28626) Spark leaves unencrypted data on local disk, even with encryption turned on (CVE-2019-10099)
[
https://issues.apache.org/jira/browse/SPARK-28626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Imran Rashid resolved SPARK-28626.
--
Resolution: Fixed
> Spark leaves unencrypted data on local disk, even with encryption turned on
> (CVE-2019-10099)
>
>
> Key: SPARK-28626
> URL: https://issues.apache.org/jira/browse/SPARK-28626
> Project: Spark
> Issue Type: Bug
> Components: Security
>Affects Versions: 2.3.2
>Reporter: Imran Rashid
>Priority: Major
> Fix For: 2.4.0, 2.3.3
>
>
> Severity: Important
>
> Vendor: The Apache Software Foundation
>
> Versions affected:
> All Spark 1.x, Spark 2.0.x, Spark 2.1.x, and 2.2.x versions
> Spark 2.3.0 to 2.3.2
>
> Description:
> Prior to Spark 2.3.3, in certain situations Spark would write user data to
> local disk unencrypted, even if spark.io.encryption.enabled=true. This
> includes cached blocks that are fetched to disk (controlled by
> spark.maxRemoteBlockSizeFetchToMem); in SparkR, using parallelize; in
> Pyspark, using broadcast and parallelize; and use of python udfs.
>
>
> Mitigation:
> 1.x, 2.0.x, 2.1.x, 2.2.x, 2.3.x users should upgrade to 2.3.3 or newer
> 2.3.x users should upgrade to 2.3.3 or newer.
>
> Credit:
> This issue was reported by Thomas Graves of NVIDIA.
>
> References:
> [https://spark.apache.org/security.html]
>
> The following commits were used to fix this issue, in branch-2.3 (there may
> be other commits in master / branch-2.4, that are equivalent.)
> {noformat}
> commit 575fea120e25249716e3f680396580c5f9e26b5b
> Author: Imran Rashid
> Date: Wed Aug 22 16:38:28 2018 -0500
> [CORE] Updates to remote cache reads
> Covered by tests in DistributedSuite
>
> commit 6d742d1bd71aa3803dce91a830b37284cb18cf70
> Author: Imran Rashid
> Date: Thu Sep 6 12:11:47 2018 -0500
> [PYSPARK][SQL] Updates to RowQueue
> Tested with updates to RowQueueSuite
>
> commit 09dd34cb1706f2477a89174d6a1a0f17ed5b0a65
> Author: Imran Rashid
> Date: Mon Aug 13 21:35:34 2018 -0500
> [PYSPARK] Updates to pyspark broadcast
>
> commit 12717ba0edfa5459c9ac2085f46b1ecc0ee759aa
> Author: hyukjinkwon
> Date: Mon Sep 24 19:25:02 2018 +0800
> [SPARKR] Match pyspark features in SparkR communication protocol
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-28592) Mark new Shuffle apis as @Experimental (instead of @Private)
[
https://issues.apache.org/jira/browse/SPARK-28592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Imran Rashid updated SPARK-28592:
-
Description: The new Shuffle api is initially marked as {{@Private}}, just
to discourage anyone from trying to make use of it before all the pieces are in
place (in particular if spark 3.0 is released before everything is merged).
But once its all merged we can change to {{@Experimental}} (was: The api is
initially marked as {{@Private}}, just to discourage anyone from trying to make
use of it before all the pieces are in place (in particular if spark 3.0 is
released before everything is merged). But once its all merged we can change
to {{@Experimental}})
> Mark new Shuffle apis as @Experimental (instead of @Private)
>
>
> Key: SPARK-28592
> URL: https://issues.apache.org/jira/browse/SPARK-28592
> Project: Spark
> Issue Type: Sub-task
> Components: Shuffle, Spark Core
>Affects Versions: 3.0.0
>Reporter: Imran Rashid
>Priority: Major
>
> The new Shuffle api is initially marked as {{@Private}}, just to discourage
> anyone from trying to make use of it before all the pieces are in place (in
> particular if spark 3.0 is released before everything is merged). But once
> its all merged we can change to {{@Experimental}}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-28592) Mark new Shuffle apis as @Experimental (instead of @Private)
[
https://issues.apache.org/jira/browse/SPARK-28592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Imran Rashid updated SPARK-28592:
-
Summary: Mark new Shuffle apis as @Experimental (instead of @Private)
(was: Mark apis as @Experimental (instead of @Private))
> Mark new Shuffle apis as @Experimental (instead of @Private)
>
>
> Key: SPARK-28592
> URL: https://issues.apache.org/jira/browse/SPARK-28592
> Project: Spark
> Issue Type: Sub-task
> Components: Shuffle, Spark Core
>Affects Versions: 3.0.0
>Reporter: Imran Rashid
>Priority: Major
>
> The api is initially marked as {{@Private}}, just to discourage anyone from
> trying to make use of it before all the pieces are in place (in particular if
> spark 3.0 is released before everything is merged). But once its all merged
> we can change to {{@Experimental}}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-26329) ExecutorMetrics should poll faster than heartbeats
[ https://issues.apache.org/jira/browse/SPARK-26329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-26329. -- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23767 [https://github.com/apache/spark/pull/23767] > ExecutorMetrics should poll faster than heartbeats > -- > > Key: SPARK-26329 > URL: https://issues.apache.org/jira/browse/SPARK-26329 > Project: Spark > Issue Type: Improvement > Components: Spark Core, Web UI >Affects Versions: 3.0.0 >Reporter: Imran Rashid >Assignee: Wing Yew Poon >Priority: Major > Fix For: 3.0.0 > > > We should allow faster polling of the executor memory metrics (SPARK-23429 / > SPARK-23206) without requiring a faster heartbeat rate. We've seen the > memory usage of executors pike over 1 GB in less than a second, but > heartbeats are only every 10 seconds (by default). Spark needs to enable > fast polling to capture these peaks, without causing too much strain on the > system. > In the current implementation, the metrics are polled along with the > heartbeat, but this leads to a slow rate of polling metrics by default. If > users were to increase the rate of the heartbeat, they risk overloading the > driver on a large cluster, with too many messages and too much work to > aggregate the metrics. But, the executor could poll the metrics more > frequently, and still only send the *max* since the last heartbeat for each > metric. This keeps the load on the driver the same, and only introduces a > small overhead on the executor to grab the metrics and keep the max. > The downside of this approach is that we still need to wait for the next > heartbeat for the driver to be aware of the new peak. If the executor dies > or is killed before then, then we won't find out. A potential future > enhancement would be to send an update *anytime* there is an increase by some > percentage, but we'll leave that out for now. > Another possibility would be to change the metrics themselves to track peaks > for us, so we don't have to fine-tune the polling rate. For example, some > jvm metrics provide a usage threshold, and notification: > https://docs.oracle.com/javase/7/docs/api/java/lang/management/MemoryPoolMXBean.html#UsageThreshold > But, that is not available on all metrics. This proposal gives us a generic > way to get a more accurate peak memory usage for *all* metrics. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-26329) ExecutorMetrics should poll faster than heartbeats
[ https://issues.apache.org/jira/browse/SPARK-26329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reassigned SPARK-26329: Assignee: Wing Yew Poon > ExecutorMetrics should poll faster than heartbeats > -- > > Key: SPARK-26329 > URL: https://issues.apache.org/jira/browse/SPARK-26329 > Project: Spark > Issue Type: Improvement > Components: Spark Core, Web UI >Affects Versions: 3.0.0 >Reporter: Imran Rashid >Assignee: Wing Yew Poon >Priority: Major > > We should allow faster polling of the executor memory metrics (SPARK-23429 / > SPARK-23206) without requiring a faster heartbeat rate. We've seen the > memory usage of executors pike over 1 GB in less than a second, but > heartbeats are only every 10 seconds (by default). Spark needs to enable > fast polling to capture these peaks, without causing too much strain on the > system. > In the current implementation, the metrics are polled along with the > heartbeat, but this leads to a slow rate of polling metrics by default. If > users were to increase the rate of the heartbeat, they risk overloading the > driver on a large cluster, with too many messages and too much work to > aggregate the metrics. But, the executor could poll the metrics more > frequently, and still only send the *max* since the last heartbeat for each > metric. This keeps the load on the driver the same, and only introduces a > small overhead on the executor to grab the metrics and keep the max. > The downside of this approach is that we still need to wait for the next > heartbeat for the driver to be aware of the new peak. If the executor dies > or is killed before then, then we won't find out. A potential future > enhancement would be to send an update *anytime* there is an increase by some > percentage, but we'll leave that out for now. > Another possibility would be to change the metrics themselves to track peaks > for us, so we don't have to fine-tune the polling rate. For example, some > jvm metrics provide a usage threshold, and notification: > https://docs.oracle.com/javase/7/docs/api/java/lang/management/MemoryPoolMXBean.html#UsageThreshold > But, that is not available on all metrics. This proposal gives us a generic > way to get a more accurate peak memory usage for *all* metrics. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-28592) Mark apis as @Experimental (instead of @Private)
Imran Rashid created SPARK-28592:
Summary: Mark apis as @Experimental (instead of @Private)
Key: SPARK-28592
URL: https://issues.apache.org/jira/browse/SPARK-28592
Project: Spark
Issue Type: Sub-task
Components: Shuffle, Spark Core
Affects Versions: 3.0.0
Reporter: Imran Rashid
The api is initially marked as {{@Private}}, just to discourage anyone from
trying to make use of it before all the pieces are in place (in particular if
spark 3.0 is released before everything is merged). But once its all merged we
can change to {{@Experimental}}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-26755) Optimize Spark Scheduler to dequeue speculative tasks more efficiently
[
https://issues.apache.org/jira/browse/SPARK-26755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Imran Rashid reassigned SPARK-26755:
Assignee: Parth Gandhi
> Optimize Spark Scheduler to dequeue speculative tasks more efficiently
> --
>
> Key: SPARK-26755
> URL: https://issues.apache.org/jira/browse/SPARK-26755
> Project: Spark
> Issue Type: Improvement
> Components: Scheduler
>Affects Versions: 3.0.0
>Reporter: Parth Gandhi
>Assignee: Parth Gandhi
>Priority: Minor
> Attachments: Screen Shot 2019-01-28 at 11.21.05 AM.png, Screen Shot
> 2019-01-28 at 11.21.25 AM.png, Screen Shot 2019-01-28 at 11.22.42 AM.png
>
>
> Currently, Spark Scheduler takes quite some time to dequeue speculative tasks
> for larger tasksets within a stage(like 10 or more) when speculation is
> turned on. On further analysis, it was found that the "task-result-getter"
> threads remain blocked on one of the dispatcher-event-loop threads holding
> the lock on TaskSchedulerImpl object
> {code:java}
> def resourceOffers(offers: IndexedSeq[WorkerOffer]):
> Seq[Seq[TaskDescription]] = synchronized {
> {code}
> which takes quite some time to execute the method "dequeueSpeculativeTask"
> in TaskSetManager.scala, thus, slowing down the overall running time of the
> spark job. We were monitoring the time utilization of that lock for the whole
> duration of the job and it was close to 50% i.e. the code within the
> synchronized block would run for almost half the duration of the entire spark
> job. The screenshots of the thread dump have been attached below for
> reference.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-26755) Optimize Spark Scheduler to dequeue speculative tasks more efficiently
[
https://issues.apache.org/jira/browse/SPARK-26755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Imran Rashid resolved SPARK-26755.
--
Resolution: Fixed
Fix Version/s: 3.0.0
Issue resolved by pull request 23677
[https://github.com/apache/spark/pull/23677]
> Optimize Spark Scheduler to dequeue speculative tasks more efficiently
> --
>
> Key: SPARK-26755
> URL: https://issues.apache.org/jira/browse/SPARK-26755
> Project: Spark
> Issue Type: Improvement
> Components: Scheduler
>Affects Versions: 3.0.0
>Reporter: Parth Gandhi
>Assignee: Parth Gandhi
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: Screen Shot 2019-01-28 at 11.21.05 AM.png, Screen Shot
> 2019-01-28 at 11.21.25 AM.png, Screen Shot 2019-01-28 at 11.22.42 AM.png
>
>
> Currently, Spark Scheduler takes quite some time to dequeue speculative tasks
> for larger tasksets within a stage(like 10 or more) when speculation is
> turned on. On further analysis, it was found that the "task-result-getter"
> threads remain blocked on one of the dispatcher-event-loop threads holding
> the lock on TaskSchedulerImpl object
> {code:java}
> def resourceOffers(offers: IndexedSeq[WorkerOffer]):
> Seq[Seq[TaskDescription]] = synchronized {
> {code}
> which takes quite some time to execute the method "dequeueSpeculativeTask"
> in TaskSetManager.scala, thus, slowing down the overall running time of the
> spark job. We were monitoring the time utilization of that lock for the whole
> duration of the job and it was close to 50% i.e. the code within the
> synchronized block would run for almost half the duration of the entire spark
> job. The screenshots of the thread dump have been attached below for
> reference.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27812) kubernetes client import non-daemon thread which block jvm exit.
[ https://issues.apache.org/jira/browse/SPARK-27812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16886359#comment-16886359 ] Imran Rashid commented on SPARK-27812: -- as mentioned elsewhere, installing an uncaught exception handler on the driver is not great because it can mess with usercode in the driver. Maybe we can make it configurable – by default, only install one if there is no uncaught exception handler already installed? No great solution here ... > kubernetes client import non-daemon thread which block jvm exit. > > > Key: SPARK-27812 > URL: https://issues.apache.org/jira/browse/SPARK-27812 > Project: Spark > Issue Type: Bug > Components: Kubernetes >Affects Versions: 2.4.3 >Reporter: Henry Yu >Priority: Major > > I try spark-submit to k8s with cluster mode. Driver pod failed to exit with > An Okhttp Websocket Non-Daemon Thread. > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-28165) SHS does not delete old inprogress files until cleaner.maxAge after SHS start time
[
https://issues.apache.org/jira/browse/SPARK-28165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873614#comment-16873614
]
Imran Rashid commented on SPARK-28165:
--
btw if anybody wants to investigate this more, here's a simple test case,
(though as discussed above, we can't just use the modtime as its not totally
trustworthy):
{code}
test("log cleaner for inprogress files before SHS startup") {
val firstFileModifiedTime = TimeUnit.SECONDS.toMillis(10)
val secondFileModifiedTime = TimeUnit.SECONDS.toMillis(100)
val maxAge = TimeUnit.SECONDS.toMillis(40)
val clock = new ManualClock(0)
val log1 = newLogFile("inProgressApp1", None, inProgress = true)
writeFile(log1, true, None,
SparkListenerApplicationStart(
"inProgressApp1", Some("inProgressApp1"), 3L, "test", Some("attempt1"))
)
log1.setLastModified(firstFileModifiedTime)
val log2 = newLogFile("inProgressApp2", None, inProgress = true)
writeFile(log2, true, None,
SparkListenerApplicationStart(
"inProgressApp2", Some("inProgressApp2"), 23L, "test2",
Some("attempt2"))
)
log2.setLastModified(secondFileModifiedTime)
// advance the clock so the first log is expired, but second log is still
recent
clock.setTime(secondFileModifiedTime)
assert(clock.getTimeMillis() > firstFileModifiedTime + maxAge)
// start up the SHS
val provider = new FsHistoryProvider(
createTestConf().set("spark.history.fs.cleaner.maxAge", s"${maxAge}ms"),
clock)
provider.checkForLogs()
// We should cleanup one log immediately
updateAndCheck(provider) { list =>
assert(list.size === 1)
}
assert(!log1.exists())
assert(log2.exists())
}
{code}
> SHS does not delete old inprogress files until cleaner.maxAge after SHS start
> time
> --
>
> Key: SPARK-28165
> URL: https://issues.apache.org/jira/browse/SPARK-28165
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.3.3, 2.4.3
>Reporter: Imran Rashid
>Priority: Major
>
> The SHS will not delete inprogress files until
> {{spark.history.fs.cleaner.maxAge}} time after it has started (7 days by
> default), regardless of when the last modification to the file was. This is
> particularly problematic if the SHS gets restarted regularly, as then you'll
> end up never deleting old files.
> There might not be much we can do about this -- we can't really trust the
> modification time of the file, as that isn't always updated reliably.
> We could take the last time of any event from the file, but then we'd have to
> turn off the optimization of SPARK-6951, to avoid reading the entire file
> just for the listing.
> *WORKAROUND*: have the SHS save state across restarts to local disk by
> specifying a path in {{spark.history.store.path}}. It'll still take 7 days
> from when you add that config for the cleaning to happen, but then going for
> the cleaning should happen reliably.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-28005) SparkRackResolver should not log for resolving empty list
[
https://issues.apache.org/jira/browse/SPARK-28005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Imran Rashid reassigned SPARK-28005:
Assignee: Gabor Somogyi
> SparkRackResolver should not log for resolving empty list
> -
>
> Key: SPARK-28005
> URL: https://issues.apache.org/jira/browse/SPARK-28005
> Project: Spark
> Issue Type: Bug
> Components: Scheduler
>Affects Versions: 3.0.0
>Reporter: Imran Rashid
>Assignee: Gabor Somogyi
>Priority: Major
> Fix For: 3.0.0
>
>
> After SPARK-13704, {{SparkRackResolver}} generates an INFO message everytime
> is called with 0 arguments:
> https://github.com/apache/spark/blob/master/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/SparkRackResolver.scala#L73-L76
> That actually happens every 1s when there are no active executors, because of
> the repeated offers that happen as part of delay scheduling:
> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala#L134-L139
> while this is relatively benign, its a pretty annoying thing to be logging at
> INFO level every 1 second.
> This is easy to reproduce -- in spark-shell, with dynamic allocation, set log
> level to info, see the logs appear every 1 second. Then run something, see
> the msgs stop. After the executors timeout, see the msgs reappear.
> {noformat}
> scala> :paste
> // Entering paste mode (ctrl-D to finish)
> sc.setLogLevel("info")
> Thread.sleep(5000)
> sc.parallelize(1 to 10).count()
> // Exiting paste mode, now interpreting.
> 19/06/11 12:43:40 INFO yarn.SparkRackResolver: Got an error when resolving
> hostNames. Falling back to /default-rack for all
> 19/06/11 12:43:41 INFO yarn.SparkRackResolver: Got an error when resolving
> hostNames. Falling back to /default-rack for all
> 19/06/11 12:43:42 INFO yarn.SparkRackResolver: Got an error when resolving
> hostNames. Falling back to /default-rack for all
> 19/06/11 12:43:43 INFO yarn.SparkRackResolver: Got an error when resolving
> hostNames. Falling back to /default-rack for all
> 19/06/11 12:43:44 INFO yarn.SparkRackResolver: Got an error when resolving
> hostNames. Falling back to /default-rack for all
> 19/06/11 12:43:45 INFO spark.SparkContext: Starting job: count at :28
> 19/06/11 12:43:45 INFO scheduler.DAGScheduler: Got job 0 (count at
> :28) with 2 output partitions
> 19/06/11 12:43:45 INFO scheduler.DAGScheduler: Final stage: ResultStage 0
> (count at :28)
> ...
> 19/06/11 12:43:54 INFO cluster.YarnScheduler: Removed TaskSet 0.0, whose
> tasks have all completed, from pool
> 19/06/11 12:43:54 INFO scheduler.DAGScheduler: ResultStage 0 (count at
> :28) finished in 9.548 s
> 19/06/11 12:43:54 INFO scheduler.DAGScheduler: Job 0 finished: count at
> :28, took 9.613049 s
> res2: Long = 10
>
> scala>
> ...
> 19/06/11 12:44:56 INFO yarn.SparkRackResolver: Got an error when resolving
> hostNames. Falling back to /default-rack for all
> 19/06/11 12:44:57 INFO yarn.SparkRackResolver: Got an error when resolving
> hostNames. Falling back to /default-rack for all
> 19/06/11 12:44:58 INFO yarn.SparkRackResolver: Got an error when resolving
> hostNames. Falling back to /default-rack for all
> 19/06/11 12:44:59 INFO yarn.SparkRackResolver: Got an error when resolving
> hostNames. Falling back to /default-rack for all
> ...
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-28005) SparkRackResolver should not log for resolving empty list
[
https://issues.apache.org/jira/browse/SPARK-28005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Imran Rashid resolved SPARK-28005.
--
Resolution: Fixed
Fix Version/s: 3.0.0
Issue resolved by pull request 24935
[https://github.com/apache/spark/pull/24935]
> SparkRackResolver should not log for resolving empty list
> -
>
> Key: SPARK-28005
> URL: https://issues.apache.org/jira/browse/SPARK-28005
> Project: Spark
> Issue Type: Bug
> Components: Scheduler
>Affects Versions: 3.0.0
>Reporter: Imran Rashid
>Priority: Major
> Fix For: 3.0.0
>
>
> After SPARK-13704, {{SparkRackResolver}} generates an INFO message everytime
> is called with 0 arguments:
> https://github.com/apache/spark/blob/master/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/SparkRackResolver.scala#L73-L76
> That actually happens every 1s when there are no active executors, because of
> the repeated offers that happen as part of delay scheduling:
> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala#L134-L139
> while this is relatively benign, its a pretty annoying thing to be logging at
> INFO level every 1 second.
> This is easy to reproduce -- in spark-shell, with dynamic allocation, set log
> level to info, see the logs appear every 1 second. Then run something, see
> the msgs stop. After the executors timeout, see the msgs reappear.
> {noformat}
> scala> :paste
> // Entering paste mode (ctrl-D to finish)
> sc.setLogLevel("info")
> Thread.sleep(5000)
> sc.parallelize(1 to 10).count()
> // Exiting paste mode, now interpreting.
> 19/06/11 12:43:40 INFO yarn.SparkRackResolver: Got an error when resolving
> hostNames. Falling back to /default-rack for all
> 19/06/11 12:43:41 INFO yarn.SparkRackResolver: Got an error when resolving
> hostNames. Falling back to /default-rack for all
> 19/06/11 12:43:42 INFO yarn.SparkRackResolver: Got an error when resolving
> hostNames. Falling back to /default-rack for all
> 19/06/11 12:43:43 INFO yarn.SparkRackResolver: Got an error when resolving
> hostNames. Falling back to /default-rack for all
> 19/06/11 12:43:44 INFO yarn.SparkRackResolver: Got an error when resolving
> hostNames. Falling back to /default-rack for all
> 19/06/11 12:43:45 INFO spark.SparkContext: Starting job: count at :28
> 19/06/11 12:43:45 INFO scheduler.DAGScheduler: Got job 0 (count at
> :28) with 2 output partitions
> 19/06/11 12:43:45 INFO scheduler.DAGScheduler: Final stage: ResultStage 0
> (count at :28)
> ...
> 19/06/11 12:43:54 INFO cluster.YarnScheduler: Removed TaskSet 0.0, whose
> tasks have all completed, from pool
> 19/06/11 12:43:54 INFO scheduler.DAGScheduler: ResultStage 0 (count at
> :28) finished in 9.548 s
> 19/06/11 12:43:54 INFO scheduler.DAGScheduler: Job 0 finished: count at
> :28, took 9.613049 s
> res2: Long = 10
>
> scala>
> ...
> 19/06/11 12:44:56 INFO yarn.SparkRackResolver: Got an error when resolving
> hostNames. Falling back to /default-rack for all
> 19/06/11 12:44:57 INFO yarn.SparkRackResolver: Got an error when resolving
> hostNames. Falling back to /default-rack for all
> 19/06/11 12:44:58 INFO yarn.SparkRackResolver: Got an error when resolving
> hostNames. Falling back to /default-rack for all
> 19/06/11 12:44:59 INFO yarn.SparkRackResolver: Got an error when resolving
> hostNames. Falling back to /default-rack for all
> ...
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-27630) Stage retry causes totalRunningTasks calculation to be negative
[
https://issues.apache.org/jira/browse/SPARK-27630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Imran Rashid resolved SPARK-27630.
--
Resolution: Fixed
Fix Version/s: 3.0.0
Issue resolved by pull request 24497
[https://github.com/apache/spark/pull/24497]
> Stage retry causes totalRunningTasks calculation to be negative
> ---
>
> Key: SPARK-27630
> URL: https://issues.apache.org/jira/browse/SPARK-27630
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.3.0
>Reporter: dzcxzl
>Assignee: dzcxzl
>Priority: Minor
> Fix For: 3.0.0
>
>
> Track tasks separately for each stage attempt (instead of tracking by stage),
> and do NOT reset the numRunningTasks to 0 on StageCompleted.
> In the case of stage retry, the {{taskEnd}} event from the zombie stage
> sometimes makes the number of {{totalRunningTasks}} negative, which will
> causes the job to get stuck.
> Similar problem also exists with {{stageIdToTaskIndices}} &
> {{stageIdToSpeculativeTaskIndices}}.
> If it is a failed {{taskEnd}} event of the zombie stage, this will cause
> {{stageIdToTaskIndices}} or {{stageIdToSpeculativeTaskIndices}} to remove the
> task index of the active stage, and the number of {{totalPendingTasks}} will
> increase unexpectedly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-27630) Stage retry causes totalRunningTasks calculation to be negative
[
https://issues.apache.org/jira/browse/SPARK-27630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Imran Rashid reassigned SPARK-27630:
Assignee: dzcxzl
> Stage retry causes totalRunningTasks calculation to be negative
> ---
>
> Key: SPARK-27630
> URL: https://issues.apache.org/jira/browse/SPARK-27630
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.3.0
>Reporter: dzcxzl
>Assignee: dzcxzl
>Priority: Minor
>
> Track tasks separately for each stage attempt (instead of tracking by stage),
> and do NOT reset the numRunningTasks to 0 on StageCompleted.
> In the case of stage retry, the {{taskEnd}} event from the zombie stage
> sometimes makes the number of {{totalRunningTasks}} negative, which will
> causes the job to get stuck.
> Similar problem also exists with {{stageIdToTaskIndices}} &
> {{stageIdToSpeculativeTaskIndices}}.
> If it is a failed {{taskEnd}} event of the zombie stage, this will cause
> {{stageIdToTaskIndices}} or {{stageIdToSpeculativeTaskIndices}} to remove the
> task index of the active stage, and the number of {{totalPendingTasks}} will
> increase unexpectedly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-28165) SHS does not delete old inprogress files until cleaner.maxAge after SHS start time
Imran Rashid created SPARK-28165:
Summary: SHS does not delete old inprogress files until
cleaner.maxAge after SHS start time
Key: SPARK-28165
URL: https://issues.apache.org/jira/browse/SPARK-28165
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 2.4.3, 2.3.3
Reporter: Imran Rashid
The SHS will not delete inprogress files until
{{spark.history.fs.cleaner.maxAge}} time after it has started (7 days by
default), regardless of when the last modification to the file was. This is
particularly problematic if the SHS gets restarted regularly, as then you'll
end up never deleting old files.
There might not be much we can do about this -- we can't really trust the
modification time of the file, as that isn't always updated reliably.
We could take the last time of any event from the file, but then we'd have to
turn off the optimization of SPARK-6951, to avoid reading the entire file just
for the listing.
*WORKAROUND*: have the SHS save state across restarts to local disk by
specifying a path in {{spark.history.store.path}}. It'll still take 7 days
from when you add that config for the cleaning to happen, but then going for
the cleaning should happen reliably.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-28005) SparkRackResolver should not log for resolving empty list
[
https://issues.apache.org/jira/browse/SPARK-28005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16861410#comment-16861410
]
Imran Rashid commented on SPARK-28005:
--
cc [~cltlfcjin]
> SparkRackResolver should not log for resolving empty list
> -
>
> Key: SPARK-28005
> URL: https://issues.apache.org/jira/browse/SPARK-28005
> Project: Spark
> Issue Type: Bug
> Components: Scheduler
>Affects Versions: 3.0.0
>Reporter: Imran Rashid
>Priority: Major
>
> After SPARK-13704, {{SparkRackResolver}} generates an INFO message everytime
> is called with 0 arguments:
> https://github.com/apache/spark/blob/master/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/SparkRackResolver.scala#L73-L76
> That actually happens every 1s when there are no active executors, because of
> the repeated offers that happen as part of delay scheduling:
> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala#L134-L139
> while this is relatively benign, its a pretty annoying thing to be logging at
> INFO level every 1 second.
> This is easy to reproduce -- in spark-shell, with dynamic allocation, set log
> level to info, see the logs appear every 1 second. Then run something, see
> the msgs stop. After the executors timeout, see the msgs reappear.
> {noformat}
> scala> :paste
> // Entering paste mode (ctrl-D to finish)
> sc.setLogLevel("info")
> Thread.sleep(5000)
> sc.parallelize(1 to 10).count()
> // Exiting paste mode, now interpreting.
> 19/06/11 12:43:40 INFO yarn.SparkRackResolver: Got an error when resolving
> hostNames. Falling back to /default-rack for all
> 19/06/11 12:43:41 INFO yarn.SparkRackResolver: Got an error when resolving
> hostNames. Falling back to /default-rack for all
> 19/06/11 12:43:42 INFO yarn.SparkRackResolver: Got an error when resolving
> hostNames. Falling back to /default-rack for all
> 19/06/11 12:43:43 INFO yarn.SparkRackResolver: Got an error when resolving
> hostNames. Falling back to /default-rack for all
> 19/06/11 12:43:44 INFO yarn.SparkRackResolver: Got an error when resolving
> hostNames. Falling back to /default-rack for all
> 19/06/11 12:43:45 INFO spark.SparkContext: Starting job: count at :28
> 19/06/11 12:43:45 INFO scheduler.DAGScheduler: Got job 0 (count at
> :28) with 2 output partitions
> 19/06/11 12:43:45 INFO scheduler.DAGScheduler: Final stage: ResultStage 0
> (count at :28)
> ...
> 19/06/11 12:43:54 INFO cluster.YarnScheduler: Removed TaskSet 0.0, whose
> tasks have all completed, from pool
> 19/06/11 12:43:54 INFO scheduler.DAGScheduler: ResultStage 0 (count at
> :28) finished in 9.548 s
> 19/06/11 12:43:54 INFO scheduler.DAGScheduler: Job 0 finished: count at
> :28, took 9.613049 s
> res2: Long = 10
>
> scala>
> ...
> 19/06/11 12:44:56 INFO yarn.SparkRackResolver: Got an error when resolving
> hostNames. Falling back to /default-rack for all
> 19/06/11 12:44:57 INFO yarn.SparkRackResolver: Got an error when resolving
> hostNames. Falling back to /default-rack for all
> 19/06/11 12:44:58 INFO yarn.SparkRackResolver: Got an error when resolving
> hostNames. Falling back to /default-rack for all
> 19/06/11 12:44:59 INFO yarn.SparkRackResolver: Got an error when resolving
> hostNames. Falling back to /default-rack for all
> ...
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-28005) SparkRackResolver should not log for resolving empty list
Imran Rashid created SPARK-28005:
Summary: SparkRackResolver should not log for resolving empty list
Key: SPARK-28005
URL: https://issues.apache.org/jira/browse/SPARK-28005
Project: Spark
Issue Type: Bug
Components: Scheduler
Affects Versions: 3.0.0
Reporter: Imran Rashid
After SPARK-13704, {{SparkRackResolver}} generates an INFO message everytime is
called with 0 arguments:
https://github.com/apache/spark/blob/master/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/SparkRackResolver.scala#L73-L76
That actually happens every 1s when there are no active executors, because of
the repeated offers that happen as part of delay scheduling:
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala#L134-L139
while this is relatively benign, its a pretty annoying thing to be logging at
INFO level every 1 second.
This is easy to reproduce -- in spark-shell, with dynamic allocation, set log
level to info, see the logs appear every 1 second. Then run something, see the
msgs stop. After the executors timeout, see the msgs reappear.
{noformat}
scala> :paste
// Entering paste mode (ctrl-D to finish)
sc.setLogLevel("info")
Thread.sleep(5000)
sc.parallelize(1 to 10).count()
// Exiting paste mode, now interpreting.
19/06/11 12:43:40 INFO yarn.SparkRackResolver: Got an error when resolving
hostNames. Falling back to /default-rack for all
19/06/11 12:43:41 INFO yarn.SparkRackResolver: Got an error when resolving
hostNames. Falling back to /default-rack for all
19/06/11 12:43:42 INFO yarn.SparkRackResolver: Got an error when resolving
hostNames. Falling back to /default-rack for all
19/06/11 12:43:43 INFO yarn.SparkRackResolver: Got an error when resolving
hostNames. Falling back to /default-rack for all
19/06/11 12:43:44 INFO yarn.SparkRackResolver: Got an error when resolving
hostNames. Falling back to /default-rack for all
19/06/11 12:43:45 INFO spark.SparkContext: Starting job: count at :28
19/06/11 12:43:45 INFO scheduler.DAGScheduler: Got job 0 (count at :28)
with 2 output partitions
19/06/11 12:43:45 INFO scheduler.DAGScheduler: Final stage: ResultStage 0
(count at :28)
...
19/06/11 12:43:54 INFO cluster.YarnScheduler: Removed TaskSet 0.0, whose tasks
have all completed, from pool
19/06/11 12:43:54 INFO scheduler.DAGScheduler: ResultStage 0 (count at
:28) finished in 9.548 s
19/06/11 12:43:54 INFO scheduler.DAGScheduler: Job 0 finished: count at
:28, took 9.613049 s
res2: Long = 10
scala>
...
19/06/11 12:44:56 INFO yarn.SparkRackResolver: Got an error when resolving
hostNames. Falling back to /default-rack for all
19/06/11 12:44:57 INFO yarn.SparkRackResolver: Got an error when resolving
hostNames. Falling back to /default-rack for all
19/06/11 12:44:58 INFO yarn.SparkRackResolver: Got an error when resolving
hostNames. Falling back to /default-rack for all
19/06/11 12:44:59 INFO yarn.SparkRackResolver: Got an error when resolving
hostNames. Falling back to /default-rack for all
...
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-20286) dynamicAllocation.executorIdleTimeout is ignored after unpersist
[
https://issues.apache.org/jira/browse/SPARK-20286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Imran Rashid resolved SPARK-20286.
--
Resolution: Fixed
Fix Version/s: 3.0.0
Issue resolved by pull request 24704
[https://github.com/apache/spark/pull/24704]
> dynamicAllocation.executorIdleTimeout is ignored after unpersist
>
>
> Key: SPARK-20286
> URL: https://issues.apache.org/jira/browse/SPARK-20286
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.0.1
>Reporter: Miguel Pérez
>Priority: Major
> Fix For: 3.0.0
>
>
> With dynamic allocation enabled, it seems that executors with cached data
> which are unpersisted are still being killed using the
> {{dynamicAllocation.cachedExecutorIdleTimeout}} configuration, instead of
> {{dynamicAllocation.executorIdleTimeout}}. Assuming the default configuration
> ({{dynamicAllocation.cachedExecutorIdleTimeout = Infinity}}), an executor
> with unpersisted data won't be released until the job ends.
> *How to reproduce*
> - Set different values for {{dynamicAllocation.executorIdleTimeout}} and
> {{dynamicAllocation.cachedExecutorIdleTimeout}}
> - Load a file into a RDD and persist it
> - Execute an action on the RDD (like a count) so some executors are activated.
> - When the action has finished, unpersist the RDD
> - The application UI removes correctly the persisted data from the *Storage*
> tab, but if you look in the *Executors* tab, you will find that the executors
> remain *active* until ({{dynamicAllocation.cachedExecutorIdleTimeout}} is
> reached.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-20286) dynamicAllocation.executorIdleTimeout is ignored after unpersist
[
https://issues.apache.org/jira/browse/SPARK-20286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Imran Rashid reassigned SPARK-20286:
Assignee: Marcelo Vanzin
> dynamicAllocation.executorIdleTimeout is ignored after unpersist
>
>
> Key: SPARK-20286
> URL: https://issues.apache.org/jira/browse/SPARK-20286
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.0.1
>Reporter: Miguel Pérez
>Assignee: Marcelo Vanzin
>Priority: Major
> Fix For: 3.0.0
>
>
> With dynamic allocation enabled, it seems that executors with cached data
> which are unpersisted are still being killed using the
> {{dynamicAllocation.cachedExecutorIdleTimeout}} configuration, instead of
> {{dynamicAllocation.executorIdleTimeout}}. Assuming the default configuration
> ({{dynamicAllocation.cachedExecutorIdleTimeout = Infinity}}), an executor
> with unpersisted data won't be released until the job ends.
> *How to reproduce*
> - Set different values for {{dynamicAllocation.executorIdleTimeout}} and
> {{dynamicAllocation.cachedExecutorIdleTimeout}}
> - Load a file into a RDD and persist it
> - Execute an action on the RDD (like a count) so some executors are activated.
> - When the action has finished, unpersist the RDD
> - The application UI removes correctly the persisted data from the *Storage*
> tab, but if you look in the *Executors* tab, you will find that the executors
> remain *active* until ({{dynamicAllocation.cachedExecutorIdleTimeout}} is
> reached.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-27780) Shuffle server & client should be versioned to enable smoother upgrade
Imran Rashid created SPARK-27780: Summary: Shuffle server & client should be versioned to enable smoother upgrade Key: SPARK-27780 URL: https://issues.apache.org/jira/browse/SPARK-27780 Project: Spark Issue Type: New Feature Components: Shuffle, Spark Core Affects Versions: 3.0.0 Reporter: Imran Rashid The external shuffle service is often upgraded at a different time than spark itself. However, this causes problems when the protocol changes between the shuffle service and the spark runtime -- this forces users to upgrade everything simultaneously. We should add versioning to the shuffle client & server, so they know what messages the other will support. This would allow better handling of mixed versions, from better error msgs to allowing some mismatched versions (with reduced capabilities). This originally came up in a discussion here: https://github.com/apache/spark/pull/24565#issuecomment-493496466 There are a few ways we could do the versioning which we still need to discuss: 1) Version specified by config. This allows for mixed versions across the cluster and rolling upgrades. It also will let a spark 3.0 client talk to a 2.4 shuffle service. But, may be a nuisance for users to get this right. 2) Auto-detection during registration with local shuffle service. This makes the versioning easy for the end user, and can even handle a 2.4 shuffle service though it does not support the new versioning. However, it will not handle a rolling upgrade correctly -- if the local shuffle service has been upgraded, but other nodes in the cluster have not, it will get the version wrong. 3) Exchange versions per-connection. When a connection is opened, the server & client could first exchange messages with their versions, so they know how to continue communication after that. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-25888) Service requests for persist() blocks via external service after dynamic deallocation
[ https://issues.apache.org/jira/browse/SPARK-25888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16837632#comment-16837632 ] Imran Rashid commented on SPARK-25888: -- There is a lot of stuff listed here, so I think it makes sense to break it out into smaller chunks. I created SPARK-27677 for the first part (which corresponds to the PR attila already created). I think all of the asks here are reasonable long-term goals, but some are significantly more complicated. I'll keep thinking about the ways we can break this down to get in incremental improvement. > Service requests for persist() blocks via external service after dynamic > deallocation > - > > Key: SPARK-25888 > URL: https://issues.apache.org/jira/browse/SPARK-25888 > Project: Spark > Issue Type: New Feature > Components: Block Manager, Shuffle, YARN >Affects Versions: 2.3.2 >Reporter: Adam Kennedy >Priority: Major > > Large and highly multi-tenant Spark on YARN clusters with diverse job > execution often display terrible utilization rates (we have observed as low > as 3-7% CPU at max container allocation, but 50% CPU utilization on even a > well policed cluster is not uncommon). > As a sizing example, consider a scenario with 1,000 nodes, 50,000 cores, 250 > users and 50,000 runs of 1,000 distinct applications per week, with > predominantly Spark including a mixture of ETL, Ad Hoc tasks and PySpark > Notebook jobs (no streaming) > Utilization problems appear to be due in large part to difficulties with > persist() blocks (DISK or DISK+MEMORY) preventing dynamic deallocation. > In situations where an external shuffle service is present (which is typical > on clusters of this type) we already solve this for the shuffle block case by > offloading the IO handling of shuffle blocks to the external service, > allowing dynamic deallocation to proceed. > Allowing Executors to transfer persist() blocks to some external "shuffle" > service in a similar manner would be an enormous win for Spark multi-tenancy > as it would limit deallocation blocking scenarios to only MEMORY-only cache() > scenarios. > I'm not sure if I'm correct, but I seem to recall seeing in the original > external shuffle service commits that may have been considered at the time > but getting shuffle blocks moved to the external shuffle service was the > first priority. > With support for external persist() DISK blocks in place, we could also then > handle deallocation of DISK+MEMORY, as the memory instance could first be > dropped, changing the block to DISK only, and then further transferred to the > shuffle service. > We have tried to resolve the persist() issue via extensive user training, but > that has typically only allowed us to improve utilization of the worst > offenders (10% utilization) up to around 40-60% utilization, as the need for > persist() is often legitimate and occurs during the middle stages of a job. > In a healthy multi-tenant scenario, a large job might spool up to say 10,000 > cores, persist() data, release executors across a long tail down to 100 > cores, and then spool back up to 10,000 cores for the following stage without > impact on the persist() data. > In an ideal world, if an new executor started up on a node on which blocks > had been transferred to the shuffle service, the new executor might even be > able to "recapture" control of those blocks (if that would help with > performance in some way). > And the behavior of gradually expanding up and down several times over the > course of a job would not just improve utilization, but would allow resources > to more easily be redistributed to other jobs which start on the cluster > during the long-tail periods, which would improve multi-tenancy and bring us > closer to optimal "envy free" YARN scheduling. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-27677) Disk-persisted RDD blocks served by shuffle service, and ignored for Dynamic Allocation
Imran Rashid created SPARK-27677: Summary: Disk-persisted RDD blocks served by shuffle service, and ignored for Dynamic Allocation Key: SPARK-27677 URL: https://issues.apache.org/jira/browse/SPARK-27677 Project: Spark Issue Type: New Feature Components: Block Manager, Spark Core Affects Versions: 2.4.3 Reporter: Imran Rashid Disk-cached RDD blocks are currently unavailable after an executor is removed. However, when there is an external shuffle service, the data remains available on disk and could be served by the shuffle service. This would allow dynamic allocation to reclaim executors with only disk-cached blocks more rapidly, but still keep the cached data available. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27677) Disk-persisted RDD blocks served by shuffle service, and ignored for Dynamic Allocation
[ https://issues.apache.org/jira/browse/SPARK-27677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16837629#comment-16837629 ] Imran Rashid commented on SPARK-27677: -- I'm breaking this out of SPARK-25888. [~attilapiros] is already working on a fix for this, the current pr is https://github.com/apache/spark/pull/24499 > Disk-persisted RDD blocks served by shuffle service, and ignored for Dynamic > Allocation > --- > > Key: SPARK-27677 > URL: https://issues.apache.org/jira/browse/SPARK-27677 > Project: Spark > Issue Type: New Feature > Components: Block Manager, Spark Core >Affects Versions: 2.4.3 >Reporter: Imran Rashid >Priority: Major > > Disk-cached RDD blocks are currently unavailable after an executor is > removed. However, when there is an external shuffle service, the data > remains available on disk and could be served by the shuffle service. This > would allow dynamic allocation to reclaim executors with only disk-cached > blocks more rapidly, but still keep the cached data available. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-27590) do not consider skipped tasks when scheduling speculative tasks
[ https://issues.apache.org/jira/browse/SPARK-27590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-27590. -- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24485 [https://github.com/apache/spark/pull/24485] > do not consider skipped tasks when scheduling speculative tasks > --- > > Key: SPARK-27590 > URL: https://issues.apache.org/jira/browse/SPARK-27590 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 3.0.0 >Reporter: Wenchen Fan >Assignee: Wenchen Fan >Priority: Major > Fix For: 3.0.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27367) Faster RoaringBitmap Serialization with v0.8.0
[ https://issues.apache.org/jira/browse/SPARK-27367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16822045#comment-16822045 ] Imran Rashid commented on SPARK-27367: -- Did you change spark code as well, to use the new suggested api for serde? Or just upgrade the version of roaring bitmap? the size of the bitmap is related to how many partitions there are on the reduce side of a shufflemap task. And then the number of messages which go to the driver is related to the number of map tasks, and how many tasks are running concurrently. Some users on large clusters run with > 10K tasks on each side. > Faster RoaringBitmap Serialization with v0.8.0 > -- > > Key: SPARK-27367 > URL: https://issues.apache.org/jira/browse/SPARK-27367 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 3.0.0 >Reporter: Imran Rashid >Priority: Major > > RoaringBitmap 0.8.0 adds faster serde, but also requires us to change how we > call the serde routines slightly to take advantage of it. This is probably a > worthwhile optimization as the every shuffle map task with a large # of > partitions generates these bitmaps, and the driver especially has to > deserialize many of these messages. > See > * https://github.com/apache/spark/pull/24264#issuecomment-479675572 > * https://github.com/RoaringBitmap/RoaringBitmap/pull/325 > * https://github.com/RoaringBitmap/RoaringBitmap/issues/319 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-25422) flaky test: org.apache.spark.DistributedSuite.caching on disk, replicated (encryption = on) (with replication as stream)
[
https://issues.apache.org/jira/browse/SPARK-25422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16821178#comment-16821178
]
Imran Rashid commented on SPARK-25422:
--
[~magnusfa] handling blocks > 2GB was broken in many ways before 2.4. You'll
need to upgrade to 2.4.0 to get it work. There are no known issues with large
blocks in 2.4.0 (as far as I know, anyway).
> flaky test: org.apache.spark.DistributedSuite.caching on disk, replicated
> (encryption = on) (with replication as stream)
>
>
> Key: SPARK-25422
> URL: https://issues.apache.org/jira/browse/SPARK-25422
> Project: Spark
> Issue Type: Test
> Components: Spark Core
>Affects Versions: 2.4.0
>Reporter: Wenchen Fan
>Assignee: Imran Rashid
>Priority: Major
> Fix For: 2.4.0
>
>
> stacktrace
> {code}
> org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in
> stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0
> (TID 7, localhost, executor 1): java.io.IOException:
> org.apache.spark.SparkException: corrupt remote block broadcast_0_piece0 of
> broadcast_0: 1651574976 != 1165629262
> at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1320)
> at
> org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:207)
> at
> org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(TorrentBroadcast.scala:66)
> at
> org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.scala:66)
> at
> org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:96)
> at org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:84)
> at org.apache.spark.scheduler.Task.run(Task.scala:121)
> at
> org.apache.spark.executor.Executor$TaskRunner$$anonfun$7.apply(Executor.scala:367)
> at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1347)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:373)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.spark.SparkException: corrupt remote block
> broadcast_0_piece0 of broadcast_0: 1651574976 != 1165629262
> at
> org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1.apply$mcVI$sp(TorrentBroadcast.scala:167)
> at
> org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1.apply(TorrentBroadcast.scala:151)
> at
> org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1.apply(TorrentBroadcast.scala:151)
> at scala.collection.immutable.List.foreach(List.scala:392)
> at
> org.apache.spark.broadcast.TorrentBroadcast.org$apache$spark$broadcast$TorrentBroadcast$$readBlocks(TorrentBroadcast.scala:151)
> at
> org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1$$anonfun$apply$2.apply(TorrentBroadcast.scala:231)
> at scala.Option.getOrElse(Option.scala:121)
> at
> org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1.apply(TorrentBroadcast.scala:211)
> at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1313)
> ... 13 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-25088) Rest Server default & doc updates
[ https://issues.apache.org/jira/browse/SPARK-25088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820321#comment-16820321 ] Imran Rashid commented on SPARK-25088: -- if you're allowing unauthed rest, what is the point of auth on standard submission? For most users, they'd just think they had a secure setup with auth on standard submission, and not realize they'd left a backdoor wide open. Its not worth that security risk > Rest Server default & doc updates > - > > Key: SPARK-25088 > URL: https://issues.apache.org/jira/browse/SPARK-25088 > Project: Spark > Issue Type: Improvement > Components: Deploy, Spark Core >Affects Versions: 2.1.3, 2.2.2, 2.3.1, 2.4.0 >Reporter: Imran Rashid >Assignee: Imran Rashid >Priority: Major > Labels: release-notes > Fix For: 2.4.0 > > > The rest server could use some updates on defaults & docs, both in standalone > and mesos. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6235) Address various 2G limits
[ https://issues.apache.org/jira/browse/SPARK-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820309#comment-16820309 ] Imran Rashid commented on SPARK-6235: - [~glal14] actually this was fixed in 2.4. There was one open issue, SPARK-24936, but I just closed that as its just improving an error msg which I think isn't really worth fixing just for spark 3.0, and so also resolved this umbrella. > Address various 2G limits > - > > Key: SPARK-6235 > URL: https://issues.apache.org/jira/browse/SPARK-6235 > Project: Spark > Issue Type: Umbrella > Components: Shuffle, Spark Core >Reporter: Reynold Xin >Priority: Major > Fix For: 2.4.0 > > Attachments: SPARK-6235_Design_V0.02.pdf > > > An umbrella ticket to track the various 2G limit we have in Spark, due to the > use of byte arrays and ByteBuffers. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-6235) Address various 2G limits
[ https://issues.apache.org/jira/browse/SPARK-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-6235. - Resolution: Fixed Fix Version/s: 2.4.0 > Address various 2G limits > - > > Key: SPARK-6235 > URL: https://issues.apache.org/jira/browse/SPARK-6235 > Project: Spark > Issue Type: Umbrella > Components: Shuffle, Spark Core >Reporter: Reynold Xin >Priority: Major > Fix For: 2.4.0 > > Attachments: SPARK-6235_Design_V0.02.pdf > > > An umbrella ticket to track the various 2G limit we have in Spark, due to the > use of byte arrays and ByteBuffers. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-24936) Better error message when trying a shuffle fetch over 2 GB
[
https://issues.apache.org/jira/browse/SPARK-24936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Imran Rashid resolved SPARK-24936.
--
Resolution: Won't Fix
As we've already shipped 2.4, I think its unlikely we're going to fix this
later. I don't think we need to worry that much about spark 3.0 talking to
shuffle services < 2.2.
If anybody is motivated, feel free to submit a pr here, but I think leaving
this open is probably misleading about the status.
> Better error message when trying a shuffle fetch over 2 GB
> --
>
> Key: SPARK-24936
> URL: https://issues.apache.org/jira/browse/SPARK-24936
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
>Affects Versions: 2.4.0
>Reporter: Imran Rashid
>Priority: Major
>
> After SPARK-24297, spark will try to fetch shuffle blocks to disk if their
> over 2GB. However, this will fail with an external shuffle service running <
> spark 2.2, with an unhelpful error message like:
> {noformat}
> 18/07/26 07:15:02 WARN scheduler.TaskSetManager: Lost task 1.0 in stage 1.3
> (TID 15, xyz.com, executor 2): FetchFailed(BlockManagerId(1
> , xyz.com, 7337, None), shuffleId=0, mapId=1, reduceId=1, message=
> org.apache.spark.shuffle.FetchFailedException:
> java.lang.UnsupportedOperationException
> at
> org.apache.spark.network.server.StreamManager.openStream(StreamManager.java:60)
> at
> org.apache.spark.network.server.TransportRequestHandler.processStreamRequest(TransportRequestHandler.java:136)
> ...
> {noformat}
> We can't do anything to make the shuffle succeed, in this situation, but we
> should fail with a better error message.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-25250) Race condition with tasks running when new attempt for same stage is created leads to other task in the next attempt running on the same partition id retry multiple ti
[ https://issues.apache.org/jira/browse/SPARK-25250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818318#comment-16818318 ] Imran Rashid commented on SPARK-25250: -- I agree about opening a new jira. Wenchen discussed reverting it here: https://github.com/apache/spark/pull/24359 I agree we made a major mistake in that fix. I don't care too much about how the commits look in git, I am fine with having a revert followed by a different fix, rather than rolling it into one change. > Race condition with tasks running when new attempt for same stage is created > leads to other task in the next attempt running on the same partition id > retry multiple times > -- > > Key: SPARK-25250 > URL: https://issues.apache.org/jira/browse/SPARK-25250 > Project: Spark > Issue Type: Bug > Components: Scheduler, Spark Core >Affects Versions: 2.3.1 >Reporter: Parth Gandhi >Assignee: Parth Gandhi >Priority: Major > Fix For: 2.3.4, 2.4.1, 3.0.0 > > > We recently had a scenario where a race condition occurred when a task from > previous stage attempt just finished before new attempt for the same stage > was created due to fetch failure, so the new task created in the second > attempt on the same partition id was retrying multiple times due to > TaskCommitDenied Exception without realizing that the task in earlier attempt > was already successful. > For example, consider a task with partition id 9000 and index 9000 running in > stage 4.0. We see a fetch failure so thus, we spawn a new stage attempt 4.1. > Just within this timespan, the above task completes successfully, thus, > marking the partition id 9000 as complete for 4.0. However, as stage 4.1 has > not yet been created, the taskset info for that stage is not available to the > TaskScheduler so, naturally, the partition id 9000 has not been marked > completed for 4.1. Stage 4.1 now spawns task with index 2000 on the same > partition id 9000. This task fails due to CommitDeniedException and since, it > does not see the corresponding partition id as been marked successful, it > keeps retrying multiple times until the job finally succeeds. It doesn't > cause any job failures because the DAG scheduler is tracking the partitions > separate from the task set managers. > > Steps to Reproduce: > # Run any large job involving shuffle operation. > # When the ShuffleMap stage finishes and the ResultStage begins running, > cause this stage to throw a fetch failure exception(Try deleting certain > shuffle files on any host). > # Observe the task attempt numbers for the next stage attempt. Please note > that this issue is an intermittent one, so it might not happen all the time. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-13704) TaskSchedulerImpl.createTaskSetManager can be expensive, and result in lost executors due to blocked heartbeats
[
https://issues.apache.org/jira/browse/SPARK-13704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Imran Rashid reassigned SPARK-13704:
Assignee: Lantao Jin
> TaskSchedulerImpl.createTaskSetManager can be expensive, and result in lost
> executors due to blocked heartbeats
> ---
>
> Key: SPARK-13704
> URL: https://issues.apache.org/jira/browse/SPARK-13704
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
>Affects Versions: 1.3.1, 1.4.1, 1.5.2, 1.6.0
>Reporter: Zhong Wang
>Assignee: Lantao Jin
>Priority: Major
> Fix For: 3.0.0
>
>
> In some cases, TaskSchedulerImpl.createTaskSetManager can be expensive. For
> example, in a Yarn cluster, it may call the topology script for rack
> awareness. When submit a very large job in a very large Yarn cluster, the
> topology script may take signifiant time to run. And this blocks receiving
> executors' heartbeats, which may result in lost executors
> Stacktraces we observed which is related to this issue:
> {code}
> "dag-scheduler-event-loop" daemon prio=10 tid=0x7f8392875800 nid=0x26e8
> runnable [0x7f83576f4000]
>java.lang.Thread.State: RUNNABLE
> at java.io.FileInputStream.readBytes(Native Method)
> at java.io.FileInputStream.read(FileInputStream.java:272)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> - locked <0xf551f460> (a
> java.lang.UNIXProcess$ProcessPipeInputStream)
> at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283)
> at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177)
> - locked <0xf5529740> (a java.io.InputStreamReader)
> at java.io.InputStreamReader.read(InputStreamReader.java:184)
> at java.io.BufferedReader.fill(BufferedReader.java:154)
> at java.io.BufferedReader.read1(BufferedReader.java:205)
> at java.io.BufferedReader.read(BufferedReader.java:279)
> - locked <0xf5529740> (a java.io.InputStreamReader)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.parseExecResult(Shell.java:728)
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:524)
> at org.apache.hadoop.util.Shell.run(Shell.java:455)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
> at
> org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.runResolveCommand(ScriptBasedMapping.java:251)
> at
> org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.resolve(ScriptBasedMapping.java:188)
> at
> org.apache.hadoop.net.CachedDNSToSwitchMapping.resolve(CachedDNSToSwitchMapping.java:119)
> at
> org.apache.hadoop.yarn.util.RackResolver.coreResolve(RackResolver.java:101)
> at
> org.apache.hadoop.yarn.util.RackResolver.resolve(RackResolver.java:81)
> at
> org.apache.spark.scheduler.cluster.YarnScheduler.getRackForHost(YarnScheduler.scala:38)
> at
> org.apache.spark.scheduler.TaskSetManager$$anonfun$org$apache$spark$scheduler$TaskSetManager$$addPendingTask$1.apply(TaskSetManager.scala:210)
> at
> org.apache.spark.scheduler.TaskSetManager$$anonfun$org$apache$spark$scheduler$TaskSetManager$$addPendingTask$1.apply(TaskSetManager.scala:189)
> at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
> at
> org.apache.spark.scheduler.TaskSetManager.org$apache$spark$scheduler$TaskSetManager$$addPendingTask(TaskSetManager.scala:189)
> at
> org.apache.spark.scheduler.TaskSetManager$$anonfun$1.apply$mcVI$sp(TaskSetManager.scala:158)
> at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141)
> at
> org.apache.spark.scheduler.TaskSetManager.(TaskSetManager.scala:157)
> at
> org.apache.spark.scheduler.TaskSchedulerImpl.createTaskSetManager(TaskSchedulerImpl.scala:187)
> at
> org.apache.spark.scheduler.TaskSchedulerImpl.submitTasks(TaskSchedulerImpl.scala:161)
> - locked <0xea3b8a88> (a
> org.apache.spark.scheduler.cluster.YarnScheduler)
> at
> org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitMissingTasks(DAGScheduler.scala:872)
> at
> org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitStage(DAGScheduler.scala:778)
> at
> org.apache.spark.scheduler.DAGScheduler.handleJobSubmitted(DAGScheduler.scala:762)
> at
>
[jira] [Resolved] (SPARK-13704) TaskSchedulerImpl.createTaskSetManager can be expensive, and result in lost executors due to blocked heartbeats
[
https://issues.apache.org/jira/browse/SPARK-13704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Imran Rashid resolved SPARK-13704.
--
Resolution: Fixed
Fix Version/s: 3.0.0
Issue resolved by pull request 24245
[https://github.com/apache/spark/pull/24245]
> TaskSchedulerImpl.createTaskSetManager can be expensive, and result in lost
> executors due to blocked heartbeats
> ---
>
> Key: SPARK-13704
> URL: https://issues.apache.org/jira/browse/SPARK-13704
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
>Affects Versions: 1.3.1, 1.4.1, 1.5.2, 1.6.0
>Reporter: Zhong Wang
>Priority: Major
> Fix For: 3.0.0
>
>
> In some cases, TaskSchedulerImpl.createTaskSetManager can be expensive. For
> example, in a Yarn cluster, it may call the topology script for rack
> awareness. When submit a very large job in a very large Yarn cluster, the
> topology script may take signifiant time to run. And this blocks receiving
> executors' heartbeats, which may result in lost executors
> Stacktraces we observed which is related to this issue:
> {code}
> "dag-scheduler-event-loop" daemon prio=10 tid=0x7f8392875800 nid=0x26e8
> runnable [0x7f83576f4000]
>java.lang.Thread.State: RUNNABLE
> at java.io.FileInputStream.readBytes(Native Method)
> at java.io.FileInputStream.read(FileInputStream.java:272)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> - locked <0xf551f460> (a
> java.lang.UNIXProcess$ProcessPipeInputStream)
> at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283)
> at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177)
> - locked <0xf5529740> (a java.io.InputStreamReader)
> at java.io.InputStreamReader.read(InputStreamReader.java:184)
> at java.io.BufferedReader.fill(BufferedReader.java:154)
> at java.io.BufferedReader.read1(BufferedReader.java:205)
> at java.io.BufferedReader.read(BufferedReader.java:279)
> - locked <0xf5529740> (a java.io.InputStreamReader)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.parseExecResult(Shell.java:728)
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:524)
> at org.apache.hadoop.util.Shell.run(Shell.java:455)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
> at
> org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.runResolveCommand(ScriptBasedMapping.java:251)
> at
> org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.resolve(ScriptBasedMapping.java:188)
> at
> org.apache.hadoop.net.CachedDNSToSwitchMapping.resolve(CachedDNSToSwitchMapping.java:119)
> at
> org.apache.hadoop.yarn.util.RackResolver.coreResolve(RackResolver.java:101)
> at
> org.apache.hadoop.yarn.util.RackResolver.resolve(RackResolver.java:81)
> at
> org.apache.spark.scheduler.cluster.YarnScheduler.getRackForHost(YarnScheduler.scala:38)
> at
> org.apache.spark.scheduler.TaskSetManager$$anonfun$org$apache$spark$scheduler$TaskSetManager$$addPendingTask$1.apply(TaskSetManager.scala:210)
> at
> org.apache.spark.scheduler.TaskSetManager$$anonfun$org$apache$spark$scheduler$TaskSetManager$$addPendingTask$1.apply(TaskSetManager.scala:189)
> at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
> at
> org.apache.spark.scheduler.TaskSetManager.org$apache$spark$scheduler$TaskSetManager$$addPendingTask(TaskSetManager.scala:189)
> at
> org.apache.spark.scheduler.TaskSetManager$$anonfun$1.apply$mcVI$sp(TaskSetManager.scala:158)
> at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141)
> at
> org.apache.spark.scheduler.TaskSetManager.(TaskSetManager.scala:157)
> at
> org.apache.spark.scheduler.TaskSchedulerImpl.createTaskSetManager(TaskSchedulerImpl.scala:187)
> at
> org.apache.spark.scheduler.TaskSchedulerImpl.submitTasks(TaskSchedulerImpl.scala:161)
> - locked <0xea3b8a88> (a
> org.apache.spark.scheduler.cluster.YarnScheduler)
> at
> org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitMissingTasks(DAGScheduler.scala:872)
> at
> org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitStage(DAGScheduler.scala:778)
> at
>
[jira] [Updated] (SPARK-27389) pyspark test failures w/ "UnknownTimeZoneError: 'US/Pacific-New'"
[
https://issues.apache.org/jira/browse/SPARK-27389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Imran Rashid updated SPARK-27389:
-
Summary: pyspark test failures w/ "UnknownTimeZoneError: 'US/Pacific-New'"
(was: Odd failures w/ "UnknownTimeZoneError: 'US/Pacific-New'")
> pyspark test failures w/ "UnknownTimeZoneError: 'US/Pacific-New'"
> -
>
> Key: SPARK-27389
> URL: https://issues.apache.org/jira/browse/SPARK-27389
> Project: Spark
> Issue Type: Task
> Components: jenkins, PySpark
>Affects Versions: 3.0.0
>Reporter: Imran Rashid
>Assignee: shane knapp
>Priority: Major
>
> I've seen a few odd PR build failures w/ an error in pyspark tests about
> "UnknownTimeZoneError: 'US/Pacific-New'". eg.
> https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4688/consoleFull
> A bit of searching tells me that US/Pacific-New probably isn't really
> supposed to be a timezone at all:
> https://mm.icann.org/pipermail/tz/2009-February/015448.html
> I'm guessing that this is from some misconfiguration of jenkins. that said,
> I can't figure out what is wrong. There does seem to be a timezone entry for
> US/Pacific-New in {{/usr/share/zoneinfo/US/Pacific-New}} -- but it seems to
> be there on every amp-jenkins-worker, so I dunno what that alone would cause
> this failure sometime.
> [~shaneknapp] I am tentatively calling this a "jenkins" issue, but I might be
> totally wrong here and it is really a pyspark problem.
> Full Stack trace from the test failure:
> {noformat}
> ==
> ERROR: test_to_pandas (pyspark.sql.tests.test_dataframe.DataFrameTests)
> --
> Traceback (most recent call last):
> File
> "/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/tests/test_dataframe.py",
> line 522, in test_to_pandas
> pdf = self._to_pandas()
> File
> "/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/tests/test_dataframe.py",
> line 517, in _to_pandas
> return df.toPandas()
> File
> "/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/dataframe.py",
> line 2189, in toPandas
> _check_series_convert_timestamps_local_tz(pdf[field.name], timezone)
> File
> "/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/types.py",
> line 1891, in _check_series_convert_timestamps_local_tz
> return _check_series_convert_timestamps_localize(s, None, timezone)
> File
> "/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/types.py",
> line 1877, in _check_series_convert_timestamps_localize
> lambda ts: ts.tz_localize(from_tz,
> ambiguous=False).tz_convert(to_tz).tz_localize(None)
> File "/home/anaconda/lib/python2.7/site-packages/pandas/core/series.py",
> line 2294, in apply
> mapped = lib.map_infer(values, f, convert=convert_dtype)
> File "pandas/src/inference.pyx", line 1207, in pandas.lib.map_infer
> (pandas/lib.c:66124)
> File
> "/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/types.py",
> line 1878, in
> if ts is not pd.NaT else pd.NaT)
> File "pandas/tslib.pyx", line 649, in pandas.tslib.Timestamp.tz_convert
> (pandas/tslib.c:13923)
> File "pandas/tslib.pyx", line 407, in pandas.tslib.Timestamp.__new__
> (pandas/tslib.c:10447)
> File "pandas/tslib.pyx", line 1467, in pandas.tslib.convert_to_tsobject
> (pandas/tslib.c:27504)
> File "pandas/tslib.pyx", line 1768, in pandas.tslib.maybe_get_tz
> (pandas/tslib.c:32362)
> File "/home/anaconda/lib/python2.7/site-packages/pytz/__init__.py", line
> 178, in timezone
> raise UnknownTimeZoneError(zone)
> UnknownTimeZoneError: 'US/Pacific-New'
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-27389) Odd failures w/ "UnknownTimeZoneError: 'US/Pacific-New'"
Imran Rashid created SPARK-27389:
Summary: Odd failures w/ "UnknownTimeZoneError: 'US/Pacific-New'"
Key: SPARK-27389
URL: https://issues.apache.org/jira/browse/SPARK-27389
Project: Spark
Issue Type: Task
Components: jenkins
Affects Versions: 3.0.0
Reporter: Imran Rashid
Assignee: shane knapp
I've seen a few odd PR build failures w/ an error in pyspark tests about
"UnknownTimeZoneError: 'US/Pacific-New'". eg.
https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4688/consoleFull
A bit of searching tells me that US/Pacific-New probably isn't really supposed
to be a timezone at all:
https://mm.icann.org/pipermail/tz/2009-February/015448.html
I'm guessing that this is from some misconfiguration of jenkins. that said, I
can't figure out what is wrong. There does seem to be a timezone entry for
US/Pacific-New in {{/usr/share/zoneinfo/US/Pacific-New}} -- but it seems to be
there on every amp-jenkins-worker, so I dunno what that alone would cause this
failure sometime.
[~shaneknapp] I am tentatively calling this a "jenkins" issue, but I might be
totally wrong here and it is really a pyspark problem.
Full Stack trace from the test failure:
{noformat}
==
ERROR: test_to_pandas (pyspark.sql.tests.test_dataframe.DataFrameTests)
--
Traceback (most recent call last):
File
"/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/tests/test_dataframe.py",
line 522, in test_to_pandas
pdf = self._to_pandas()
File
"/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/tests/test_dataframe.py",
line 517, in _to_pandas
return df.toPandas()
File
"/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/dataframe.py",
line 2189, in toPandas
_check_series_convert_timestamps_local_tz(pdf[field.name], timezone)
File
"/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/types.py",
line 1891, in _check_series_convert_timestamps_local_tz
return _check_series_convert_timestamps_localize(s, None, timezone)
File
"/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/types.py",
line 1877, in _check_series_convert_timestamps_localize
lambda ts: ts.tz_localize(from_tz,
ambiguous=False).tz_convert(to_tz).tz_localize(None)
File "/home/anaconda/lib/python2.7/site-packages/pandas/core/series.py", line
2294, in apply
mapped = lib.map_infer(values, f, convert=convert_dtype)
File "pandas/src/inference.pyx", line 1207, in pandas.lib.map_infer
(pandas/lib.c:66124)
File
"/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/types.py",
line 1878, in
if ts is not pd.NaT else pd.NaT)
File "pandas/tslib.pyx", line 649, in pandas.tslib.Timestamp.tz_convert
(pandas/tslib.c:13923)
File "pandas/tslib.pyx", line 407, in pandas.tslib.Timestamp.__new__
(pandas/tslib.c:10447)
File "pandas/tslib.pyx", line 1467, in pandas.tslib.convert_to_tsobject
(pandas/tslib.c:27504)
File "pandas/tslib.pyx", line 1768, in pandas.tslib.maybe_get_tz
(pandas/tslib.c:32362)
File "/home/anaconda/lib/python2.7/site-packages/pytz/__init__.py", line 178,
in timezone
raise UnknownTimeZoneError(zone)
UnknownTimeZoneError: 'US/Pacific-New'
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-27389) Odd failures w/ "UnknownTimeZoneError: 'US/Pacific-New'"
[
https://issues.apache.org/jira/browse/SPARK-27389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Imran Rashid updated SPARK-27389:
-
Component/s: PySpark
> Odd failures w/ "UnknownTimeZoneError: 'US/Pacific-New'"
>
>
> Key: SPARK-27389
> URL: https://issues.apache.org/jira/browse/SPARK-27389
> Project: Spark
> Issue Type: Task
> Components: jenkins, PySpark
>Affects Versions: 3.0.0
>Reporter: Imran Rashid
>Assignee: shane knapp
>Priority: Major
>
> I've seen a few odd PR build failures w/ an error in pyspark tests about
> "UnknownTimeZoneError: 'US/Pacific-New'". eg.
> https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4688/consoleFull
> A bit of searching tells me that US/Pacific-New probably isn't really
> supposed to be a timezone at all:
> https://mm.icann.org/pipermail/tz/2009-February/015448.html
> I'm guessing that this is from some misconfiguration of jenkins. that said,
> I can't figure out what is wrong. There does seem to be a timezone entry for
> US/Pacific-New in {{/usr/share/zoneinfo/US/Pacific-New}} -- but it seems to
> be there on every amp-jenkins-worker, so I dunno what that alone would cause
> this failure sometime.
> [~shaneknapp] I am tentatively calling this a "jenkins" issue, but I might be
> totally wrong here and it is really a pyspark problem.
> Full Stack trace from the test failure:
> {noformat}
> ==
> ERROR: test_to_pandas (pyspark.sql.tests.test_dataframe.DataFrameTests)
> --
> Traceback (most recent call last):
> File
> "/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/tests/test_dataframe.py",
> line 522, in test_to_pandas
> pdf = self._to_pandas()
> File
> "/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/tests/test_dataframe.py",
> line 517, in _to_pandas
> return df.toPandas()
> File
> "/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/dataframe.py",
> line 2189, in toPandas
> _check_series_convert_timestamps_local_tz(pdf[field.name], timezone)
> File
> "/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/types.py",
> line 1891, in _check_series_convert_timestamps_local_tz
> return _check_series_convert_timestamps_localize(s, None, timezone)
> File
> "/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/types.py",
> line 1877, in _check_series_convert_timestamps_localize
> lambda ts: ts.tz_localize(from_tz,
> ambiguous=False).tz_convert(to_tz).tz_localize(None)
> File "/home/anaconda/lib/python2.7/site-packages/pandas/core/series.py",
> line 2294, in apply
> mapped = lib.map_infer(values, f, convert=convert_dtype)
> File "pandas/src/inference.pyx", line 1207, in pandas.lib.map_infer
> (pandas/lib.c:66124)
> File
> "/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/types.py",
> line 1878, in
> if ts is not pd.NaT else pd.NaT)
> File "pandas/tslib.pyx", line 649, in pandas.tslib.Timestamp.tz_convert
> (pandas/tslib.c:13923)
> File "pandas/tslib.pyx", line 407, in pandas.tslib.Timestamp.__new__
> (pandas/tslib.c:10447)
> File "pandas/tslib.pyx", line 1467, in pandas.tslib.convert_to_tsobject
> (pandas/tslib.c:27504)
> File "pandas/tslib.pyx", line 1768, in pandas.tslib.maybe_get_tz
> (pandas/tslib.c:32362)
> File "/home/anaconda/lib/python2.7/site-packages/pytz/__init__.py", line
> 178, in timezone
> raise UnknownTimeZoneError(zone)
> UnknownTimeZoneError: 'US/Pacific-New'
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-27216) Upgrade RoaringBitmap to 0.7.45 to fix Kryo unsafe ser/dser issue
[
https://issues.apache.org/jira/browse/SPARK-27216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Imran Rashid resolved SPARK-27216.
--
Resolution: Fixed
Assignee: Lantao Jin
> Upgrade RoaringBitmap to 0.7.45 to fix Kryo unsafe ser/dser issue
> -
>
> Key: SPARK-27216
> URL: https://issues.apache.org/jira/browse/SPARK-27216
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.3.3, 2.4.0, 3.0.0
>Reporter: Lantao Jin
>Assignee: Lantao Jin
>Priority: Major
> Fix For: 3.0.0
>
>
> HighlyCompressedMapStatus uses RoaringBitmap to record the empty blocks. But
> RoaringBitmap-0.5.11 couldn't be ser/deser with unsafe KryoSerializer.
> We can use below UT to reproduce:
> {code}
> test("kryo serialization with RoaringBitmap") {
> val bitmap = new RoaringBitmap
> bitmap.add(1787)
> val safeSer = new KryoSerializer(conf).newInstance()
> val bitmap2 : RoaringBitmap =
> safeSer.deserialize(safeSer.serialize(bitmap))
> assert(bitmap2.equals(bitmap))
> conf.set("spark.kryo.unsafe", "true")
> val unsafeSer = new KryoSerializer(conf).newInstance()
> val bitmap3 : RoaringBitmap =
> unsafeSer.deserialize(unsafeSer.serialize(bitmap))
> assert(bitmap3.equals(bitmap)) // this will fail
> }
> {code}
> Upgrade to latest version 0.7.45 to fix it
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27216) Upgrade RoaringBitmap to 0.7.45 to fix Kryo unsafe ser/dser issue
[
https://issues.apache.org/jira/browse/SPARK-27216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809415#comment-16809415
]
Imran Rashid commented on SPARK-27216:
--
Fixed by https://github.com/apache/spark/pull/24264 in master / 3.0.0
> Upgrade RoaringBitmap to 0.7.45 to fix Kryo unsafe ser/dser issue
> -
>
> Key: SPARK-27216
> URL: https://issues.apache.org/jira/browse/SPARK-27216
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.3.3, 2.4.0, 3.0.0
>Reporter: Lantao Jin
>Assignee: Lantao Jin
>Priority: Major
> Fix For: 3.0.0
>
>
> HighlyCompressedMapStatus uses RoaringBitmap to record the empty blocks. But
> RoaringBitmap-0.5.11 couldn't be ser/deser with unsafe KryoSerializer.
> We can use below UT to reproduce:
> {code}
> test("kryo serialization with RoaringBitmap") {
> val bitmap = new RoaringBitmap
> bitmap.add(1787)
> val safeSer = new KryoSerializer(conf).newInstance()
> val bitmap2 : RoaringBitmap =
> safeSer.deserialize(safeSer.serialize(bitmap))
> assert(bitmap2.equals(bitmap))
> conf.set("spark.kryo.unsafe", "true")
> val unsafeSer = new KryoSerializer(conf).newInstance()
> val bitmap3 : RoaringBitmap =
> unsafeSer.deserialize(unsafeSer.serialize(bitmap))
> assert(bitmap3.equals(bitmap)) // this will fail
> }
> {code}
> Upgrade to latest version 0.7.45 to fix it
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-27216) Upgrade RoaringBitmap to 0.7.45 to fix Kryo unsafe ser/dser issue
[
https://issues.apache.org/jira/browse/SPARK-27216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Imran Rashid updated SPARK-27216:
-
Fix Version/s: 3.0.0
> Upgrade RoaringBitmap to 0.7.45 to fix Kryo unsafe ser/dser issue
> -
>
> Key: SPARK-27216
> URL: https://issues.apache.org/jira/browse/SPARK-27216
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.3.3, 2.4.0, 3.0.0
>Reporter: Lantao Jin
>Priority: Major
> Fix For: 3.0.0
>
>
> HighlyCompressedMapStatus uses RoaringBitmap to record the empty blocks. But
> RoaringBitmap-0.5.11 couldn't be ser/deser with unsafe KryoSerializer.
> We can use below UT to reproduce:
> {code}
> test("kryo serialization with RoaringBitmap") {
> val bitmap = new RoaringBitmap
> bitmap.add(1787)
> val safeSer = new KryoSerializer(conf).newInstance()
> val bitmap2 : RoaringBitmap =
> safeSer.deserialize(safeSer.serialize(bitmap))
> assert(bitmap2.equals(bitmap))
> conf.set("spark.kryo.unsafe", "true")
> val unsafeSer = new KryoSerializer(conf).newInstance()
> val bitmap3 : RoaringBitmap =
> unsafeSer.deserialize(unsafeSer.serialize(bitmap))
> assert(bitmap3.equals(bitmap)) // this will fail
> }
> {code}
> Upgrade to latest version 0.7.45 to fix it
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-27367) Faster RoaringBitmap Serialization with v0.8.0
Imran Rashid created SPARK-27367: Summary: Faster RoaringBitmap Serialization with v0.8.0 Key: SPARK-27367 URL: https://issues.apache.org/jira/browse/SPARK-27367 Project: Spark Issue Type: Improvement Components: Spark Core Affects Versions: 3.0.0 Reporter: Imran Rashid RoaringBitmap 0.8.0 adds faster serde, but also requires us to change how we call the serde routines slightly to take advantage of it. This is probably a worthwhile optimization as the every shuffle map task with a large # of partitions generates these bitmaps, and the driver especially has to deserialize many of these messages. See * https://github.com/apache/spark/pull/24264#issuecomment-479675572 * https://github.com/RoaringBitmap/RoaringBitmap/pull/325 * https://github.com/RoaringBitmap/RoaringBitmap/issues/319 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27220) Remove Yarn specific leftover from CoarseGrainedSchedulerBackend
[
https://issues.apache.org/jira/browse/SPARK-27220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16808000#comment-16808000
]
Imran Rashid commented on SPARK-27220:
--
{quote}
because we add something to executor id and it ends up with the necessity to
copy whole CoarseGrainedSchedulerBackend
{quote}
so you have a custom fork of the cluster manager, which generates non-int
executor ids? This isn't really a public extension point for spark, and you
also really haven't explained why would generate something different. But if
there really is a good reason, and the fix is small, you can post a patch --
just be aware this is pretty low priority.
> Remove Yarn specific leftover from CoarseGrainedSchedulerBackend
>
>
> Key: SPARK-27220
> URL: https://issues.apache.org/jira/browse/SPARK-27220
> Project: Spark
> Issue Type: Task
> Components: Spark Core, YARN
>Affects Versions: 2.0.2, 2.1.3, 2.2.3, 2.3.3, 2.4.0
>Reporter: Jacek Lewandowski
>Priority: Minor
>
> {{CoarseGrainedSchedulerBackend}} has the following field:
> {code:scala}
> // The num of current max ExecutorId used to re-register appMaster
> @volatile protected var currentExecutorIdCounter = 0
> {code}
> which is then updated:
> {code:scala}
> case RegisterExecutor(executorId, executorRef, hostname, cores,
> logUrls) =>
> ...
> // This must be synchronized because variables mutated
> // in this block are read when requesting executors
> CoarseGrainedSchedulerBackend.this.synchronized {
> executorDataMap.put(executorId, data)
> if (currentExecutorIdCounter < executorId.toInt) {
> currentExecutorIdCounter = executorId.toInt
> }
> ...
> {code}
> However it is never really used in {{CoarseGrainedSchedulerBackend}}. Its
> only usage is in Yarn-specific code. It should be moved to Yarn then because
> {{executorId}} is a {{String}} and there are really no guarantees that it is
> always an integer. It was introduced in SPARK-12864
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27216) Upgrade RoaringBitmap to 0.7.45 to fix Kryo unsafe ser/dser issue
[
https://issues.apache.org/jira/browse/SPARK-27216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807749#comment-16807749
]
Imran Rashid commented on SPARK-27216:
--
I think its probably fine to upgrade it, I just wanted to check if we knew what
the issue was.
Though spark only uses this library in one spot, its a pretty important one --
managing MapStatus is a major source of pressure on the driver on large
clusters.
> Upgrade RoaringBitmap to 0.7.45 to fix Kryo unsafe ser/dser issue
> -
>
> Key: SPARK-27216
> URL: https://issues.apache.org/jira/browse/SPARK-27216
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.3.3, 2.4.0, 3.0.0
>Reporter: Lantao Jin
>Priority: Major
>
> HighlyCompressedMapStatus uses RoaringBitmap to record the empty blocks. But
> RoaringBitmap-0.5.11 couldn't be ser/deser with unsafe KryoSerializer.
> We can use below UT to reproduce:
> {code}
> test("kryo serialization with RoaringBitmap") {
> val bitmap = new RoaringBitmap
> bitmap.add(1787)
> val safeSer = new KryoSerializer(conf).newInstance()
> val bitmap2 : RoaringBitmap =
> safeSer.deserialize(safeSer.serialize(bitmap))
> assert(bitmap2.equals(bitmap))
> conf.set("spark.kryo.unsafe", "true")
> val unsafeSer = new KryoSerializer(conf).newInstance()
> val bitmap3 : RoaringBitmap =
> unsafeSer.deserialize(unsafeSer.serialize(bitmap))
> assert(bitmap3.equals(bitmap)) // this will fail
> }
> {code}
> Upgrade to latest version 0.7.45 to fix it
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27216) Upgrade RoaringBitmap to 0.7.45 to fix Kryo unsafe ser/dser issue
[
https://issues.apache.org/jira/browse/SPARK-27216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807181#comment-16807181
]
Imran Rashid commented on SPARK-27216:
--
Any chance you know what the issue in roaring bitmap is? Just wondering as it
is a pretty large version bump.
BTW, 0.7.45 also seems to have some big performance improvements
(https://github.com/RoaringBitmap/RoaringBitmap/pull/320) for deserialization,
it might be worth considering changing spark to take advantage of that
(definitely a separate issue, might not even matter for the way spark uses
roaring bitmap anyway ...)
> Upgrade RoaringBitmap to 0.7.45 to fix Kryo unsafe ser/dser issue
> -
>
> Key: SPARK-27216
> URL: https://issues.apache.org/jira/browse/SPARK-27216
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.3.3, 2.4.0, 3.0.0
>Reporter: Lantao Jin
>Priority: Major
>
> HighlyCompressedMapStatus uses RoaringBitmap to record the empty blocks. But
> RoaringBitmap-0.5.11 couldn't be ser/deser with unsafe KryoSerializer.
> We can use below UT to reproduce:
> {code}
> test("kryo serialization with RoaringBitmap") {
> val bitmap = new RoaringBitmap
> bitmap.add(1787)
> val safeSer = new KryoSerializer(conf).newInstance()
> val bitmap2 : RoaringBitmap =
> safeSer.deserialize(safeSer.serialize(bitmap))
> assert(bitmap2.equals(bitmap))
> conf.set("spark.kryo.unsafe", "true")
> val unsafeSer = new KryoSerializer(conf).newInstance()
> val bitmap3 : RoaringBitmap =
> unsafeSer.deserialize(unsafeSer.serialize(bitmap))
> assert(bitmap3.equals(bitmap)) // this will fail
> }
> {code}
> Upgrade to latest version 0.7.45 to fix it
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27121) Resolve Scala compiler failure for Java 9+ in REPL
[
https://issues.apache.org/jira/browse/SPARK-27121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16804184#comment-16804184
]
Imran Rashid commented on SPARK-27121:
--
I think this might just be an issue with the test setup. When I try to just
run the spark-shell using java 11, things seem to work fine (the same things
that break in the tests).
There is a bunch of code in those tests which is trying to be smart w/ the
classpath to run the repl in the same jvm:
https://github.com/apache/spark/blob/50cded590f8c16ba3263a5ecba6805fb06dd8a64/repl/src/test/scala/org/apache/spark/repl/ReplSuite.scala#L40-L56
> Resolve Scala compiler failure for Java 9+ in REPL
> --
>
> Key: SPARK-27121
> URL: https://issues.apache.org/jira/browse/SPARK-27121
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Shell
>Affects Versions: 3.0.0
>Reporter: Sean Owen
>Priority: Major
>
> Currently under Java 11:
> {code}
> SingletonReplSuite:
> [init] error: error while loading Object, Missing dependency 'object scala in
> compiler mirror', required by /modules/java.base/java/lang/Object.class
> Failed to initialize compiler: object scala in compiler mirror not found.
> ** Note that as of 2.8 scala does not assume use of the java classpath.
> ** For the old behavior pass -usejavacp to scala, or if using a Settings
> ** object programmatically, settings.usejavacp.value = true.
> Failed to initialize compiler: object scala in compiler mirror not found.
> ** Note that as of 2.8 scala does not assume use of the java classpath.
> ** For the old behavior pass -usejavacp to scala, or if using a Settings
> ** object programmatically, settings.usejavacp.value = true.
> org.apache.spark.repl.SingletonReplSuite *** ABORTED ***
> Exception encountered when invoking run on a nested suite - The code passed
> to eventually never returned normally. Attempted 110 times over 50.062409015
> seconds. Last failure message: cond.apply() was false current output:
> Interpreter encountered errors during initialization!
> . (SingletonReplSuite.scala:92)
> ReplSuite:
> - propagation of local properties
> [init] error: error while loading Object, Missing dependency 'object scala in
> compiler mirror', required by /modules/java.base/java/lang/Object.class
> ...
> {code}
> This is a fairly well-known issue with Scala and Java 9, though workarounds
> online don't seem to apply to the REPL. So far I do not think Scala promises
> compiler compatibility with Java 9+, and indeed that's why we compile with
> Java 8. However the REPL compiles on the fly.
> See for example https://github.com/sbt/sbt/issues/2958
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27220) Remove Yarn specific leftover from CoarseGrainedSchedulerBackend
[
https://issues.apache.org/jira/browse/SPARK-27220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16802202#comment-16802202
]
Imran Rashid commented on SPARK-27220:
--
sorry for the delay. i don't have a really strong opinion here. It does seem
that field is only used by yarn. But because its getting updated inside the
handling of {{RegisterExecutor}}, its kind of a headache to extract.
I actually think executorIds are really always ints -- that field is only a
string because there are places the string "driver" is used as an "executorId".
Eg. in standalone mode
https://github.com/apache/spark/blob/05168e725d2a17c4164ee5f9aa068801ec2454f4/core/src/main/scala/org/apache/spark/deploy/master/ApplicationInfo.scala#L70
I feel like this is too minor to bother with, really ...
> Remove Yarn specific leftover from CoarseGrainedSchedulerBackend
>
>
> Key: SPARK-27220
> URL: https://issues.apache.org/jira/browse/SPARK-27220
> Project: Spark
> Issue Type: Task
> Components: Spark Core, YARN
>Affects Versions: 2.0.2, 2.1.3, 2.2.3, 2.3.3, 2.4.0
>Reporter: Jacek Lewandowski
>Priority: Minor
>
> {{CoarseGrainedSchedulerBackend}} has the following field:
> {code:scala}
> // The num of current max ExecutorId used to re-register appMaster
> @volatile protected var currentExecutorIdCounter = 0
> {code}
> which is then updated:
> {code:scala}
> case RegisterExecutor(executorId, executorRef, hostname, cores,
> logUrls) =>
> ...
> // This must be synchronized because variables mutated
> // in this block are read when requesting executors
> CoarseGrainedSchedulerBackend.this.synchronized {
> executorDataMap.put(executorId, data)
> if (currentExecutorIdCounter < executorId.toInt) {
> currentExecutorIdCounter = executorId.toInt
> }
> ...
> {code}
> However it is never really used in {{CoarseGrainedSchedulerBackend}}. Its
> only usage is in Yarn-specific code. It should be moved to Yarn then because
> {{executorId}} is a {{String}} and there are really no guarantees that it is
> always an integer. It was introduced in SPARK-12864
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21097) Dynamic allocation will preserve cached data
[
https://issues.apache.org/jira/browse/SPARK-21097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16801960#comment-16801960
]
Imran Rashid commented on SPARK-21097:
--
{quote}
I'm wondering if this is going to be subsumed by the Shuffle Service redesign
proposal.
{quote}
The shuffle service redesign does not subsume this. It only covers shuffle
blocks currently. While there are some commonalities in shuffle vs. cached
blocks, there are also enough differences they need special handling.
You could *extend* the redesigned shuffle service to do something like this,
but it wouldn't be exactly the same. You'd be giving up locality on the
executors if you move cached rdd blocks over, so it may be tricky to decide
when to move blocks over. It would definitely be additional work we'd do on
top of the shuffle service (which I think might make the most sense as the way
to do this in any case).
> Dynamic allocation will preserve cached data
>
>
> Key: SPARK-21097
> URL: https://issues.apache.org/jira/browse/SPARK-21097
> Project: Spark
> Issue Type: Improvement
> Components: Block Manager, Scheduler, Spark Core
>Affects Versions: 2.2.0, 2.3.0
>Reporter: Brad
>Priority: Major
> Attachments: Preserving Cached Data with Dynamic Allocation.pdf
>
>
> We want to use dynamic allocation to distribute resources among many notebook
> users on our spark clusters. One difficulty is that if a user has cached data
> then we are either prevented from de-allocating any of their executors, or we
> are forced to drop their cached data, which can lead to a bad user experience.
> We propose adding a feature to preserve cached data by copying it to other
> executors before de-allocation. This behavior would be enabled by a simple
> spark config. Now when an executor reaches its configured idle timeout,
> instead of just killing it on the spot, we will stop sending it new tasks,
> replicate all of its rdd blocks onto other executors, and then kill it. If
> there is an issue while we replicate the data, like an error, it takes too
> long, or there isn't enough space, then we will fall back to the original
> behavior and drop the data and kill the executor.
> This feature should allow anyone with notebook users to use their cluster
> resources more efficiently. Also since it will be completely opt-in it will
> unlikely to cause problems for other use cases.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27112) Spark Scheduler encounters two independent Deadlocks when trying to kill executors either due to dynamic allocation or blacklisting
[
https://issues.apache.org/jira/browse/SPARK-27112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16797321#comment-16797321
]
Imran Rashid commented on SPARK-27112:
--
[~Dhruve Ashar] -- rc8 is already defined, there is nothing I (or anybody else)
can do to change that. I simply updated the jira to reflect that. However,
you might request that rc8 does not become 2.4.1, and instead we roll an rc9
with this this fix. You should respond to the VOTE thread for rc8 on the dev
list with your concerns, that's the right forum for this (thanks for the
reminder btw, I will mention it there as well).
> Spark Scheduler encounters two independent Deadlocks when trying to kill
> executors either due to dynamic allocation or blacklisting
>
>
> Key: SPARK-27112
> URL: https://issues.apache.org/jira/browse/SPARK-27112
> Project: Spark
> Issue Type: Bug
> Components: Scheduler, Spark Core
>Affects Versions: 2.4.0, 3.0.0
>Reporter: Parth Gandhi
>Assignee: Parth Gandhi
>Priority: Major
> Fix For: 2.3.4, 2.4.2, 3.0.0
>
> Attachments: Screen Shot 2019-02-26 at 4.10.26 PM.png, Screen Shot
> 2019-02-26 at 4.10.48 PM.png, Screen Shot 2019-02-26 at 4.11.11 PM.png,
> Screen Shot 2019-02-26 at 4.11.26 PM.png
>
>
> Recently, a few spark users in the organization have reported that their jobs
> were getting stuck. On further analysis, it was found out that there exist
> two independent deadlocks and either of them occur under different
> circumstances. The screenshots for these two deadlocks are attached here.
> We were able to reproduce the deadlocks with the following piece of code:
>
> {code:java}
> import org.apache.hadoop.conf.Configuration
> import org.apache.hadoop.fs.{FileSystem, Path}
> import org.apache.spark._
> import org.apache.spark.TaskContext
> // Simple example of Word Count in Scala
> object ScalaWordCount {
> def main(args: Array[String]) {
> if (args.length < 2) {
> System.err.println("Usage: ScalaWordCount ")
> System.exit(1)
> }
> val conf = new SparkConf().setAppName("Scala Word Count")
> val sc = new SparkContext(conf)
> // get the input file uri
> val inputFilesUri = args(0)
> // get the output file uri
> val outputFilesUri = args(1)
> while (true) {
> val textFile = sc.textFile(inputFilesUri)
> val counts = textFile.flatMap(line => line.split(" "))
> .map(word => {if (TaskContext.get.partitionId == 5 &&
> TaskContext.get.attemptNumber == 0) throw new Exception("Fail for
> blacklisting") else (word, 1)})
> .reduceByKey(_ + _)
> counts.saveAsTextFile(outputFilesUri)
> val conf: Configuration = new Configuration()
> val path: Path = new Path(outputFilesUri)
> val hdfs: FileSystem = FileSystem.get(conf)
> hdfs.delete(path, true)
> }
> sc.stop()
> }
> }
> {code}
>
> Additionally, to ensure that the deadlock surfaces up soon enough, I also
> added a small delay in the Spark code here:
> [https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala#L256]
>
> {code:java}
> executorIdToFailureList.remove(exec)
> updateNextExpiryTime()
> Thread.sleep(2000)
> killBlacklistedExecutor(exec)
> {code}
>
> Also make sure that the following configs are set when launching the above
> spark job:
> *spark.blacklist.enabled=true*
> *spark.blacklist.killBlacklistedExecutors=true*
> *spark.blacklist.application.maxFailedTasksPerExecutor=1*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-27112) Spark Scheduler encounters two independent Deadlocks when trying to kill executors either due to dynamic allocation or blacklisting
[
https://issues.apache.org/jira/browse/SPARK-27112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Imran Rashid updated SPARK-27112:
-
Fix Version/s: (was: 2.4.1)
2.4.2
> Spark Scheduler encounters two independent Deadlocks when trying to kill
> executors either due to dynamic allocation or blacklisting
>
>
> Key: SPARK-27112
> URL: https://issues.apache.org/jira/browse/SPARK-27112
> Project: Spark
> Issue Type: Bug
> Components: Scheduler, Spark Core
>Affects Versions: 2.4.0, 3.0.0
>Reporter: Parth Gandhi
>Assignee: Parth Gandhi
>Priority: Major
> Fix For: 2.3.4, 2.4.2, 3.0.0
>
> Attachments: Screen Shot 2019-02-26 at 4.10.26 PM.png, Screen Shot
> 2019-02-26 at 4.10.48 PM.png, Screen Shot 2019-02-26 at 4.11.11 PM.png,
> Screen Shot 2019-02-26 at 4.11.26 PM.png
>
>
> Recently, a few spark users in the organization have reported that their jobs
> were getting stuck. On further analysis, it was found out that there exist
> two independent deadlocks and either of them occur under different
> circumstances. The screenshots for these two deadlocks are attached here.
> We were able to reproduce the deadlocks with the following piece of code:
>
> {code:java}
> import org.apache.hadoop.conf.Configuration
> import org.apache.hadoop.fs.{FileSystem, Path}
> import org.apache.spark._
> import org.apache.spark.TaskContext
> // Simple example of Word Count in Scala
> object ScalaWordCount {
> def main(args: Array[String]) {
> if (args.length < 2) {
> System.err.println("Usage: ScalaWordCount ")
> System.exit(1)
> }
> val conf = new SparkConf().setAppName("Scala Word Count")
> val sc = new SparkContext(conf)
> // get the input file uri
> val inputFilesUri = args(0)
> // get the output file uri
> val outputFilesUri = args(1)
> while (true) {
> val textFile = sc.textFile(inputFilesUri)
> val counts = textFile.flatMap(line => line.split(" "))
> .map(word => {if (TaskContext.get.partitionId == 5 &&
> TaskContext.get.attemptNumber == 0) throw new Exception("Fail for
> blacklisting") else (word, 1)})
> .reduceByKey(_ + _)
> counts.saveAsTextFile(outputFilesUri)
> val conf: Configuration = new Configuration()
> val path: Path = new Path(outputFilesUri)
> val hdfs: FileSystem = FileSystem.get(conf)
> hdfs.delete(path, true)
> }
> sc.stop()
> }
> }
> {code}
>
> Additionally, to ensure that the deadlock surfaces up soon enough, I also
> added a small delay in the Spark code here:
> [https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala#L256]
>
> {code:java}
> executorIdToFailureList.remove(exec)
> updateNextExpiryTime()
> Thread.sleep(2000)
> killBlacklistedExecutor(exec)
> {code}
>
> Also make sure that the following configs are set when launching the above
> spark job:
> *spark.blacklist.enabled=true*
> *spark.blacklist.killBlacklistedExecutors=true*
> *spark.blacklist.application.maxFailedTasksPerExecutor=1*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
