Ngone51 commented on a change in pull request #23871:
[SPARK-23433][SPARK-25250] [CORE] Later created TaskSet should learn about the
finished partitions
URL: https://github.com/apache/spark/pull/23871#discussion_r259582845
##########
File path:
core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
##########
@@ -837,17 +847,31 @@ private[spark] class TaskSchedulerImpl(
}
/**
- * Marks the task has completed in all TaskSetManagers for the given stage.
+ * Marks the task has completed in all TaskSetManagers(active / zombie) for
the given stage.
*
* After stage failure and retry, there may be multiple TaskSetManagers for
the stage.
* If an earlier attempt of a stage completes a task, we should ensure that
the later attempts
* do not also submit those same tasks. That also means that a task
completion from an earlier
* attempt can lead to the entire stage getting marked as successful.
+ * And there's a situation that the active TaskSetManager corresponding to
the stage may
+ * haven't been created at the time we call this method. And it is possible
since the behaviour
+ * of calling on this method and creating active TaskSetManager is from two
different threads,
+ * which are "task-result-getter" and "dag-scheduler-event-loop" separately.
Consequently, under
+ * this situation, the active TaskSetManager which is created later could
not learn about the
+ * finished partitions and keep on launching duplicate tasks, which may lead
to job fail for some
+ * severe cases, see SPARK-25250 for details. So, to avoid the problem, we
record the finished
+ * partitions for that stage here and exclude the already finished tasks
when we creating active
+ * TaskSetManagers later by looking into stageIdToFinishedPartitions. Thus,
active TaskSetManager
+ * could be always notified about the finished partitions whether it has
been created or not at
+ * the time we call this method.
Review comment:
updated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
