[
https://issues.apache.org/jira/browse/SPARK-20116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-20116.
----------------------------------
Resolution: Incomplete
> Remove task-level functionality from the DAGScheduler
> -----------------------------------------------------
>
> Key: SPARK-20116
> URL: https://issues.apache.org/jira/browse/SPARK-20116
> Project: Spark
> Issue Type: Sub-task
> Components: Scheduler
> Affects Versions: 2.2.0
> Reporter: Kay Ousterhout
> Assignee: Kay Ousterhout
> Priority: Major
> Labels: bulk-closed
>
> Long, long ago, the scheduler code was more modular, and the DAGScheduler
> handled the logic of scheduling DAGs of stages (as the name suggests) and the
> TaskSchedulerImpl handled scheduling the tasks within a stage. Over time,
> more and more task-specific functionality has been added to the DAGScheduler,
> and now, the DAGScheduler duplicates a bunch of the task tracking that's done
> by other scheduler components. This makes the scheduler code harder to
> reason about, and has led to some tricky bugs (e.g., SPARK-19263). We should
> move all of this functionality back to the TaskSchedulerImpl and
> TaskSetManager, which should "hide" that complexity from the DAGScheduler.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
