Github user mwws commented on the pull request:
https://github.com/apache/spark/pull/8760#issuecomment-142872717
Hi @squito , I do appreciate your detail review and comment, it helped a
lot to improve the code quality. I have submit the latest change based on your
comment and will add some unit test later.
About your comment about *immediately killing blacklisted executors*, to
have a clear segregation of duty, I don't think BlacklistTracker should handled
it. In my point of view, BlacklistTracker is duty to decide which executors and
nodes are problematic and provide such information to TaskScheduler and
ExecutorAllocator, ect. And the ExecutorAllocator(Yarn, Mesos, Standalnoe has
different allocator) should actually kill or create executors, and the strategy
about whether kill executor or not might be better to decide in allocator
instead. Your suggestation could be doable with dynamic allocation which adjust
the number of executors periodically, but I think it's another thing apart from
what we focus here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
