Kay Ousterhout created SPARK-20116:
--------------------------------------
Summary: Remove task-level functionality from the DAGScheduler
Key: SPARK-20116
URL: https://issues.apache.org/jira/browse/SPARK-20116
Project: Spark
Issue Type: Sub-task
Components: Scheduler
Affects Versions: 2.2.0
Reporter: Kay Ousterhout
Assignee: Kay Ousterhout
Long, long ago, the scheduler code was more modular, and the DAGScheduler
handled the logic of scheduling DAGs of stages (as the name suggests) and the
TaskSchedulerImpl handled scheduling the tasks within a stage. Over time, more
and more task-specific functionality has been added to the DAGScheduler, and
now, the DAGScheduler duplicates a bunch of the task tracking that's done by
other scheduler components. This makes the scheduler code harder to reason
about, and has led to some tricky bugs (e.g., SPARK-19263). We should move all
of this functionality back to the TaskSchedulerImpl and TaskSetManager, which
should "hide" that complexity from the DAGScheduler.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
