Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/8760#discussion_r45756368
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -636,7 +622,8 @@ private[spark] class TaskSetManager(
logInfo("Ignoring task-finished event for " + info.id + " in stage "
+ taskSet.id +
" because task " + index + " has already completed successfully")
}
- failedExecutors.remove(index)
+
+ blacklistTracker.foreach(_.updateFailureExecutors(stageId, info,
Success))
--- End diff --
there is one really annoying detail here. If there are multiple attempts
for one stage (which happens on an executor failure), `info.index` is not
referencing the same thing between different `TaskSet`s. Attempt 2 might only
need to run a subset of tasks, in which case index 7 could be partition 58. I
think what you want here is actually `tasks(index).partition`, which will refer
to the same logic task across all attempts of a stage.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
