Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/18663
I haven't had a chance to look at this yet, but this doesn't by chance fix
the allocator re-evaluating if it needs executors all the time does it?
I have seen issues where executors can idle timeout, because scheduler
isn't scheduling them fast enough, might be busy or the locality wait settings
interfere. it gets down to only a few executors even though it has 10000+ tasks
to run still. If it doesn't I will file a separate jira for that.
I'll try to review this later today
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
