Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/11528#issuecomment-193309137
I agree with the above that we shouldn't add another config for this. I
would rather see the default behavior changed in the 2.x line. If dynamic
allocation config is on then num-executors goes to max and initial # of
executors. I think this would allow users to easily cap their usage and would
still allow it to free up executors. It would also allow users doing ML start
out with a # of executors and if they are actually caching the data the
executors wouldn't be freed up. So you would get very similar behavior to if
dynamic allocation was off.
thoughts?
We have been running into this issue a lot recently on our clusters where
we see a lot of wasted resources and so the dynamic allocation would help but
users are so used to specifying --num-executors they don't end up using it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
