[jira] [Updated] (SPARK-10370) After a stages map outputs are registered, all running attempts should be marked as zombies
[
https://issues.apache.org/jira/browse/SPARK-10370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-10370:
-
Labels: bulk-closed (was: )
> After a stages map outputs are registered, all running attempts should be
> marked as zombies
> ---
>
> Key: SPARK-10370
> URL: https://issues.apache.org/jira/browse/SPARK-10370
> Project: Spark
> Issue Type: Bug
> Components: Scheduler
>Affects Versions: 1.5.0
>Reporter: Imran Rashid
>Priority: Major
> Labels: bulk-closed
>
> Follow up to SPARK-5259. During stage retry, its possible for a stage to
> "complete" by registering all its map output and starting the downstream
> stages, before the latest task set has completed. This will result in the
> earlier task set continuing to submit tasks, that are both unnecessary and
> increase the chance of hitting SPARK-8029.
> Spark should mark all tasks sets for a stage as zombie as soon as its map
> output is registered. Note that this involves coordination between the
> various scheduler components ({{DAGScheduler}} and {{TaskSetManager}} at
> least) which isn't easily testable with the current setup.
> To be clear, this is *not* just referring to canceling running tasks (which
> may be taken care of by SPARK-2666). This is to make sure that the taskset
> is marked as a zombie, to prevent submitting *new* tasks from this task set.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-10370) After a stages map outputs are registered, all running attempts should be marked as zombies
[
https://issues.apache.org/jira/browse/SPARK-10370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Imran Rashid updated SPARK-10370:
-
Component/s: (was: Spark Core)
Scheduler
> After a stages map outputs are registered, all running attempts should be
> marked as zombies
> ---
>
> Key: SPARK-10370
> URL: https://issues.apache.org/jira/browse/SPARK-10370
> Project: Spark
> Issue Type: Bug
> Components: Scheduler
>Affects Versions: 1.5.0
>Reporter: Imran Rashid
>
> Follow up to SPARK-5259. During stage retry, its possible for a stage to
> "complete" by registering all its map output and starting the downstream
> stages, before the latest task set has completed. This will result in the
> earlier task set continuing to submit tasks, that are both unnecessary and
> increase the chance of hitting SPARK-8029.
> Spark should mark all tasks sets for a stage as zombie as soon as its map
> output is registered. Note that this involves coordination between the
> various scheduler components ({{DAGScheduler}} and {{TaskSetManager}} at
> least) which isn't easily testable with the current setup.
> To be clear, this is *not* just referring to canceling running tasks (which
> may be taken care of by SPARK-2666). This is to make sure that the taskset
> is marked as a zombie, to prevent submitting *new* tasks from this task set.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-10370) After a stages map outputs are registered, all running attempts should be marked as zombies
[
https://issues.apache.org/jira/browse/SPARK-10370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Imran Rashid updated SPARK-10370:
-
Description:
Follow up to SPARK-5259. During stage retry, its possible for a stage to
"complete" by registering all its map output and starting the downstream
stages, before the latest task set has completed. This will result in the
earlier task set continuing to submit tasks, that are both unnecessary and
increase the chance of hitting SPARK-8029.
Spark should mark all tasks sets for a stage as zombie as soon as its map
output is registered. Note that this involves coordination between the various
scheduler components ({{DAGScheduler}} and {{TaskSetManager}} at least) which
isn't easily testable with the current setup.
To be clear, this is *not* just referring to canceling running tasks (which may
be taken care of by SPARK-2666). This is to make sure that the taskset is
marked as a zombie, to prevent submitting *new* tasks from this task set.
was:
Follow up to SPARK-5259. During stage retry, its possible for a stage to
"complete" by registering all its map output and starting the downstream
stages, before the latest task set has completed. This will result in the
earlier task set continuing to submit tasks, that are both unnecessary and
increase the chance of hitting SPARK-8029.
Spark should mark all tasks sets for a stage as zombie as soon as its map
output is registered. Note that this involves coordination between the various
scheduler components ({{DAGScheduler}} and {{TaskSetManager}} at least) which
isn't easily testable with the current setup.
> After a stages map outputs are registered, all running attempts should be
> marked as zombies
> ---
>
> Key: SPARK-10370
> URL: https://issues.apache.org/jira/browse/SPARK-10370
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 1.5.0
>Reporter: Imran Rashid
>
> Follow up to SPARK-5259. During stage retry, its possible for a stage to
> "complete" by registering all its map output and starting the downstream
> stages, before the latest task set has completed. This will result in the
> earlier task set continuing to submit tasks, that are both unnecessary and
> increase the chance of hitting SPARK-8029.
> Spark should mark all tasks sets for a stage as zombie as soon as its map
> output is registered. Note that this involves coordination between the
> various scheduler components ({{DAGScheduler}} and {{TaskSetManager}} at
> least) which isn't easily testable with the current setup.
> To be clear, this is *not* just referring to canceling running tasks (which
> may be taken care of by SPARK-2666). This is to make sure that the taskset
> is marked as a zombie, to prevent submitting *new* tasks from this task set.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-10370) After a stages map outputs are registered, all running attempts should be marked as zombies
[
https://issues.apache.org/jira/browse/SPARK-10370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Imran Rashid updated SPARK-10370:
-
Assignee: (was: Imran Rashid)
> After a stages map outputs are registered, all running attempts should be
> marked as zombies
> ---
>
> Key: SPARK-10370
> URL: https://issues.apache.org/jira/browse/SPARK-10370
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 1.5.0
>Reporter: Imran Rashid
>
> Follow up to SPARK-5259. During stage retry, its possible for a stage to
> "complete" by registering all its map output and starting the downstream
> stages, before the latest task set has completed. This will result in the
> earlier task set continuing to submit tasks, that are both unnecessary and
> increase the chance of hitting SPARK-8029.
> Spark should mark all tasks sets for a stage as zombie as soon as its map
> output is registered. Note that this involves coordination between the
> various scheduler components ({{DAGScheduler}} and {{TaskSetManager}} at
> least) which isn't easily testable with the current setup.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
