pgandhi999 commented on a change in pull request #23927: [SPARK-23433][CORE]
avoid more than one active task set managers for a stage
URL: https://github.com/apache/spark/pull/23927#discussion_r262098472
##########
File path:
core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
##########
@@ -212,14 +212,20 @@ private[spark] class TaskSchedulerImpl(
val stage = taskSet.stageId
val stageTaskSets =
taskSetsByStageIdAndAttempt.getOrElseUpdate(stage, new HashMap[Int,
TaskSetManager])
- stageTaskSets(taskSet.stageAttemptId) = manager
- val conflictingTaskSet = stageTaskSets.exists { case (_, ts) =>
- ts.taskSet != taskSet && !ts.isZombie
- }
- if (conflictingTaskSet) {
- throw new IllegalStateException(s"more than one active taskSet for
stage $stage:" +
- s" ${stageTaskSets.toSeq.map{_._2.taskSet.id}.mkString(",")}")
+
+ // Mark all the existing TaskSetManagers of this stage as zombie, as we
are adding a new one.
+ // This is necessary to handle a corner case. Let's say a stage has 10
partitions and has 2
+ // TaskSetManagers: TSM1(zombie) and TSM2(active). TSM1 has a running
task for partition 10
+ // and it completes. TSM2 finishes tasks for partition 1-19, and thinks
he is still active
+ // because partition 10 is not completed yet. However, DAGScheduler gets
task completion
+ // events for all the 10 partitions and thinks the stage is finished. If
it's a shuffle stage
+ // and somehow it has missing map outputs, then DAGScheduler will
resubmit it and create a
+ // TSM3 for it. As a stage can't have more than one active task set
managers, we must mark
+ // TSM2 as zombie (it actually is).
Review comment:
Yep it makes sense to fix the issue that this PR addresses alongwith the
other PR's for SPARK-25250.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
