Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/5018#issuecomment-85262710
@vanzin @sryza Thanks for working on the fix. I was away for the past week
and did not have the chance to review this before it went in. Regarding the
code being overly complicated, the reason why the bookkeeping is done in each
of the three places you pointed out is the following:
- We need to do it in `ExecutorAllocationManager` so we don't keep
requesting beyond the configured maximum
- We need to do it in `CoarseGrainedExecutorBackend` because the user can
bypass the dynamic scaling logic and explicitly request executors through
`sc.requestTotalExecutors`.
- We need to do it in `YarnAllocator` to ensure we don't over-allocate
containers, regardless of whether `sc.requestTotalExecutors` or the dynamic
scaling logic is used.
My intent is also to simplify the code so as to minimize the possibility of
this feature breaking again in a future release. If either of you have concrete
suggestions on refactoring it for this purpose, I would love to hear them.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
