Github user markhamstra commented on the pull request:
https://github.com/apache/spark/pull/9154#issuecomment-149667908
> what's the point of keeping a pending list for a dead executor?
I haven't done enough code spelunking to know, but I'm wondering whether
the pending list for a dead executor may become useful for "I'm not dead"
executors. [An extended Monty Python quote is very tempting, and surprisingly
relevant, at this point.] It's entirely possible for executors to miss
heartbeats (usually because they have ingested some bad user code that is now
consuming most of their CPU resources) and appear dead to the master, only to
arise from the dead a short time later (and potentially promptly miss another
heartbeat.) If the before-you-were-dead pending task lists are or should be
reattached to the Lazurus executors, then there potentially is a point to
keeping them around.
On the other hand, the "Ah, thank you very much" approach of clubbing the
"not dead" is often what ends up needing to be done manually for these
executors that refuse to die, so trying to complete their resurrection and make
them useful again may be a fool's errand regardless.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
