GitHub user zsxwing opened a pull request: https://github.com/apache/spark/pull/22771
[SPARK-25773][Core]Cancel zombie tasks in a result stage when the job finishes ## What changes were proposed in this pull request? When a job finishes, there may be some zombie tasks still running due to stage retry. Since a result stage will never be used by other jobs, running these tasks are just wasting the cluster resource. This PR just asks TaskScheduler to cancel the running tasks of a result stage when it's already finished. Credits go to @srinathshankar who suggested this idea to me. This PR also fixes two minor issues while I'm touching DAGScheduler: - Invalid spark.job.interruptOnCancel should not crash DAGScheduler. - Non fatal errors should not crash DAGScheduler. ## How was this patch tested? The new unit tests. You can merge this pull request into a Git repository by running: $ git pull https://github.com/zsxwing/spark SPARK-25773 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/22771.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #22771 ---- commit 581ea53b57cc9fc0e89f2d635422653cfdfcb27f Author: Shixiong Zhu <shixiong@...> Date: 2018-10-16T22:07:04Z Cancel zombie tasks in a result stage when the job finishes ---- --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org