Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/19048
> This is when things get out of sync because now the scheduler will set
the number of total executors needed from 4 to 1.
Have you actually observed that behavior?
The way I understand the code, both `ExecutorAllocationManager` and
`CoarseGrainedSchedulerBackend` keep track of the target number separately, and
deal in absolutes. So you have this order of events:
- 10 executors are running
- EAM detects 5 as idle and requests that 5 be killed, update its internal
target to 5
- CGSB tries to kill 5 executors and update its internal target to 5 too
- EAM in its periodic task tells CGSB that it expects 5 executors to exist
- Everybody is happy
Is that not what you're seeing?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
