Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/15986#discussion_r89252852
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala ---
@@ -525,7 +525,12 @@ private[spark] class TaskSchedulerImpl(
* of any running tasks, since the loss reason defines whether we'll
fail those tasks.
--- End diff --
I'm a bit confused about this comment since this seems to suggest that
we'll do some sort of per-task cleanup at some later time. If anyone knows
which cleanup this is referring to then maybe we should consider not updating
`executorIdToRunningTaskIds` at all in here and instead maybe should be
performing the `taskIdToExecutorId` and `taskIdToTaskSetManager` updates
somewhere else.
On the other hand, the only place where we currently remove entries from
`taskIdToExecutorId` and `taskIdToTaskSetManager` are in `statusUpdate`, so my
hunch is that the eventual cleanup alluded to here isn't happening in
standalone mode.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
