Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/5964#discussion_r31100869
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -1103,9 +1103,14 @@ class DAGScheduler(
// multiple tasks running concurrently on different executors). In
that case, it is possible
// the fetch failure has already been handled by the scheduler.
if (runningStages.contains(failedStage)) {
- logInfo(s"Marking $failedStage (${failedStage.name}) as failed "
+
- s"due to a fetch failure from $mapStage (${mapStage.name})")
- markStageAsFinished(failedStage, Some(failureMessage))
+ if (failedStage.attemptId - 1 > task.stageAttemptId) {
--- End diff --
hmm, good point. I think it works in my existing test case, because
`submitStage` already checks if the stage is running before submitting it. So
now this makes the stage simultaneously `running` and `failed` :/. Most likely
this would result in issues if my test case had an even longer pipeline of
stages in one job, so at some point a later attempt for this stage would
succeed, so it would no longer be `running` and only be `failed`, and then it
would get resubmitted for no reason. this is just from the top of my head
though ... I'll need to look more carefully and try some more cases to see what
is going on here.
(btw, thanks for looking at it in this state, I do still plan on splitting
this apart some, just keep getting sidetracked ...)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
