Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/5018#issuecomment-85572315
@andrewor14 agreed that the bookkeeping needs to be done in these three
places. Didn't meant to imply that the code is deeply muddled. To be more
precise, there are two places that I think there may be opportunity for
simplifying the code, one caused by this patch, one in light of this patch.
* Before this patch, the bookkeeping in CoarseGrainedSchedulerBackend
wasn't used in between the ExecutorAllocationManager and the YarnAllocator.
numPendingExecutors was updated, but never read. It would be nice to go back
to basically having the ExecutorAllocationManager communicate directly with the
YarnAllocator.
* In YarnAllocator, this patch was able to remove the effect of
killExecutor on targetNumExecutors. I'm wondering whether a similar change in
ExecutorAllocationManager is possible, though this warrants further
investigation.
Will try to look deeper when I have the time.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
