Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/6750#issuecomment-121761513
@squito I think there's a simpler fix here: I don't think TaskSchedulerImpl
actually needs to know about all of the potential zombie task attempts. Right
now there are two things that happen in TaskSchedulerImpl.taskSetFinished: the
task set is removed from the set of active task sets, and the schedulable is
removed (for the purposes of fairness accounting). What if we split this into
two methods: one called taskSetNoLongerActive() that signals when a task set
has become a zombie (so can be removed from activeTaskSets) and then rename
taskSetFinished to allTasksInTaskSetFinished() or something, that removes the
schedulable. Then you could change activeTaskSets to be indexed on stage id
and have a single task attempt, as in my original suggestion. I prefer that
approach because it avoids tracking unnecessarily (and complex) state about all
of the zombie task attempts. Thoughts?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
