viirya commented on a change in pull request #23927: [SPARK-23433][CORE] avoid
more than one active task set managers for a stage
URL: https://github.com/apache/spark/pull/23927#discussion_r262387082
##########
File path:
core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
##########
@@ -212,14 +212,20 @@ private[spark] class TaskSchedulerImpl(
val stage = taskSet.stageId
val stageTaskSets =
taskSetsByStageIdAndAttempt.getOrElseUpdate(stage, new HashMap[Int,
TaskSetManager])
- stageTaskSets(taskSet.stageAttemptId) = manager
- val conflictingTaskSet = stageTaskSets.exists { case (_, ts) =>
- ts.taskSet != taskSet && !ts.isZombie
- }
- if (conflictingTaskSet) {
- throw new IllegalStateException(s"more than one active taskSet for
stage $stage:" +
- s" ${stageTaskSets.toSeq.map{_._2.taskSet.id}.mkString(",")}")
+
+ // Mark all the existing TaskSetManagers of this stage as zombie, as we
are adding a new one.
+ // This is necessary to handle a corner case. Let's say a stage has 10
partitions and has 2
+ // TaskSetManagers: TSM1(zombie) and TSM2(active). TSM1 has a running
task for partition 10
+ // and it completes. TSM2 finishes tasks for partition 1-19, and thinks
he is still active
Review comment:
typo: 1-9.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
