Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/6750#discussion_r34711221
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala ---
@@ -319,26 +337,24 @@ private[spark] class TaskSchedulerImpl(
failedExecutor = Some(execId)
}
}
- taskIdToTaskSetId.get(tid) match {
- case Some(taskSetId) =>
+ taskSetManagerForTask(tid) match {
+ case Some(taskSet) =>
if (TaskState.isFinished(state)) {
- taskIdToTaskSetId.remove(tid)
+ taskIdToStageIdAndAttempt.remove(tid)
taskIdToExecutorId.remove(tid)
}
- activeTaskSets.get(taskSetId).foreach { taskSet =>
- if (state == TaskState.FINISHED) {
- taskSet.removeRunningTask(tid)
- taskResultGetter.enqueueSuccessfulTask(taskSet, tid,
serializedData)
- } else if (Set(TaskState.FAILED, TaskState.KILLED,
TaskState.LOST).contains(state)) {
- taskSet.removeRunningTask(tid)
- taskResultGetter.enqueueFailedTask(taskSet, tid, state,
serializedData)
- }
+ if (state == TaskState.FINISHED) {
+ taskSet.removeRunningTask(tid)
+ taskResultGetter.enqueueSuccessfulTask(taskSet, tid,
serializedData)
+ } else if (Set(TaskState.FAILED, TaskState.KILLED,
TaskState.LOST).contains(state)) {
+ taskSet.removeRunningTask(tid)
+ taskResultGetter.enqueueFailedTask(taskSet, tid, state,
serializedData)
}
case None =>
logError(
--- End diff --
there is a small change in logic here. Before, we'd hit this case only if
`taskIdToTaskSetId` didn't contain `tid`. If `activeTaskSets` doesn't contain
`taskSetId`, we'd do nothing (foreach on None). But now, we hit the error case
if either `taskIdToTaskSetId` doesn't contain `tid`, or if
`taskSetsByStageIdAndAttempt` doesn't contain the stage & attempt.
This wasn't exactly intentional, it just fell out from an attempt at
refactoring a helper method -- but it does seems to more accurately correspond
to what is in the error msg. Still I'm not certain about it. Perhaps if we
nobody else has high confidence in this change I should just go back to the old
behavior.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
