Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/4008#issuecomment-69657578
To quickly trace through the current behavior:
- The ["kill" link
handler](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/ui/jobs/StagesTab.scala#L42)
calls `SparkContext.cancelStage()`.
-
[This](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/SparkContext.scala#L1436),
in turn, calls DAGScheduler.cancelStage().
- In the
[handleStageCancellation](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L1148),
handler, DAGScheduler finds all jobs that depend on a stage and cancels those
jobs.
-
[This](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L1165),
in turn,
[fails](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L1194),
the job and all stages used only by that job ("independent stages"). When
failing a stage, DAGScheduler [tries to
cancel](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L1221)
any tasks running in that stage. For a while, the Mesos scheduler backend
didn't support task cancellation, but this has been fixed. Task cancellation
is asynchronous and best-effort, but the expected behavior is that canceling a
stage will kill any running tasks in that stage.
I guess the issue here is that the task cancellation isn't sufficient to
clean up the receiver because it's run in a separate thread that's not cleaned
up when the task exits. We can't necessarily rely on threads being interrupted
when they're cancelled (see
https://github.com/apache/spark/commit/432201c7ee9e1ea1d70a6418cbad1c5ad2653ed3),
so this seems like a reasonable approach.
I have a couple of code-level comments about the approach proposed here,
though, so let me comment on those in-line.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
