Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/3525#issuecomment-73523911
I'm still not on board with having 2 different configs. You then have to
explain to the user what they are, which one takes presendence, etc. I can see
cases the % is useful but then again you can argue the other way too. If my
job needs 4G overhead, I may want to increase the executor memory without
changing the overhead amount. In that case then I have to go mess with % to
get it back what I want. Either way I think the user has to experiment to get
the best number. I haven't seen any reason to bump up the default at this
point but if others have more real world data lets consider it.
How many people are complaining or having real issues with this?
The main reason I'm against adding this is that I consider it an api change
and we now have to support 2 configs until we can get rid of the deprecated
one. Its just more dev overhead, testing, and potential user confusion. If
its enough of a benefit to the user to change it then I would be ok with it but
would rather deprecate the existing one in favor of the %. To me the raw
number is more flexible then the % but I see the % as being easier in
situations.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
