Github user sitalkedia commented on the issue:
https://github.com/apache/spark/pull/19048
>> this code in the EAM: Should be changed to account for the current
number of executors, so that the EAM doesn't tell the CGSB that it wants less
executors than currently exist.
Actually if you look at the api, `ExecutorAllocationManager` api, this is
how `requestTotalExecutors` behaves - `The total number of executors we'd like
to have. The cluster manager shouldn't kill any running executor to reach this
number, but, if all existing executors were to die, this is the number of
executors
we'd want to be allocated.` So the EAM is right in setting the number of
total executors it needs to 5 because lets say all executors die, it is up to
the cluster manager to spawn 5 executors (not 10).
>>Your solution (the new updateTotalExecutor) looks too much like the
existing replace parameter, and it's a little confusing if you try to think
about how to use both. What does it mean to ask for updateTotalExecutor = false
and replace = false? The latter means you want the executor count to go down,
while the former means you don't.
I agree with you on this. May be it would be cleaner if we provide a new
api like this - `killExecutorsAndNotUpdateTotal`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
