Github user sitalkedia commented on the issue:
https://github.com/apache/spark/pull/19048
Looking at the scheduler and the dynamic executor allocator code, this is
what my understanding, correct me if I am wrong.
Let's say the dynamic executor allocator is ramping down the number of
executors. There are 10 executors running and it needs only 4. Then
`ExecutorAllocationManager` will make a call to set the total executors to 4
and also try to kill some idle executors (say 3). This is when things get out
of sync because now the scheduler will set the number of total executors needed
from 4 to 1.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
