Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16620#discussion_r98703486
--- Diff: core/src/main/scala/org/apache/spark/scheduler/Stage.scala ---
@@ -68,6 +68,12 @@ private[scheduler] abstract class Stage(
/** Set of jobs that this stage belongs to. */
val jobIds = new HashSet[Int]
+ /**
+ * Partitions which there is not yet a task succeeded on. Note that for
[[ShuffleMapStage]]
+ * pendingPartitions.size() == 0 doesn't mean the stage is available.
Because the succeeded
+ * task can be bogus which is out of date and task's epoch is older than
corresponding
+ * executor's failed epoch in [[DAGScheduler]].
+ */
--- End diff --
How about:
Partitions the DAGScheduler is waiting on before it tries to mark the stage
/ job as completed and continue. Most commonly, this is the set of tasks that
are not successful in the active taskset for this stage, but not always. In
particular, when there are multiple attempts for a stage, then this will
include late task completions from earlier attempts. Finally, note that when
this is empty, it does not *necessarily* mean that stage is completed -- we
have may have lost some of the map output from that stage. But the
DAGScheduler will check for this condition and resubmit the stage if necessary.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
