Merge pull request #216 from liancheng/fix-spark-966 Bugfix: SPARK-965 & SPARK-966
SPARK-965: https://spark-project.atlassian.net/browse/SPARK-965 SPARK-966: https://spark-project.atlassian.net/browse/SPARK-966 * Add back `DAGScheduler.start()`, `eventProcessActor` is created and started here. Notice that function is only called by `SparkContext`. * Cancel the scheduled stage resubmission task when stopping `eventProcessActor` * Add a new `DAGSchedulerEvent` `ResubmitFailedStages` This event message is sent by the scheduled stage resubmission task to `eventProcessActor`. In this way, `DAGScheduler.resubmitFailedStages()` is guaranteed to be executed from the same thread that runs `DAGScheduler.processEvent()`. Please refer to discussion in [SPARK-966](https://spark-project.atlassian.net/browse/SPARK-966) for details. Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/60e23a58 Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/60e23a58 Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/60e23a58 Branch: refs/heads/master Commit: 60e23a58b288dae3c87da28e1506323b1d88ee9e Parents: 743a31a 4a1d966 Author: Reynold Xin <[email protected]> Authored: Sat Nov 30 23:38:49 2013 -0800 Committer: Reynold Xin <[email protected]> Committed: Sat Nov 30 23:38:49 2013 -0800 ---------------------------------------------------------------------- .../scala/org/apache/spark/SparkContext.scala | 1 + .../apache/spark/scheduler/DAGScheduler.scala | 90 ++++++++++++++------ .../spark/scheduler/DAGSchedulerEvent.scala | 2 + 3 files changed, 66 insertions(+), 27 deletions(-) ----------------------------------------------------------------------
