Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/13826#discussion_r68638551
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala ---
@@ -298,10 +298,10 @@ private[spark] class TaskSchedulerImpl(
if (!executorIdToHost.contains(o.executorId)) {
--- End diff --
Should we be checking executorIdToTaskCount here? That seems to be what we
use everywhere else to detect executor liveness. It looks like, when a new
executor is added, we always add it to both maps (here), and then when it's
removed, sometimes we remove it from executorIdToTaskCount before removing it
from executorIdToHost (if we're still waiting for the reason the executor was
lost). Since we don't reuse executor IDs, AFAIK, it doesn't actually matter
which map we check here, so might be good to do executorIdToTaskCount for
consistency (and then change line 302 to just update).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
