JoshRosen commented on code in PR #43954:
URL: https://github.com/apache/spark/pull/43954#discussion_r1402526009
##########
core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala:
##########
@@ -302,10 +302,19 @@ private[spark] class TaskSchedulerImpl(
reason: String): Unit = synchronized {
logInfo("Cancelling stage " + stageId)
// Kill all running tasks for the stage.
- killAllTaskAttempts(stageId, interruptThread, reason = "Stage cancelled: "
+ reason)
- // Cancel all attempts for the stage.
+ logInfo(s"Killing all running tasks in stage $stageId: $reason")
taskSetsByStageIdAndAttempt.get(stageId).foreach { attempts =>
attempts.foreach { case (_, tsm) =>
+ // There are two possible cases here:
+ // 1. The task set manager has been created and some tasks have been
scheduled.
+ // In this case, send a kill signal to the executors to kill the
task.
+ // 2. The task set manager has been created but no tasks have been
scheduled. In this case,
+ // simply continue.
+ tsm.runningTasksSet.foreach { tid =>
+ taskIdToExecutorId.get(tid).foreach { execId =>
+ backend.killTask(tid, execId, interruptThread, reason)
Review Comment:
In order to preserve the old behaviors for the task kill reason, should this
be
```
reason = "Stage cancelled: " + reason
```
instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
