bnetzi commented on PR #40771:
URL: https://github.com/apache/spark/pull/40771#issuecomment-2619648271
I know this PR is closed for a while but I think the decision not to merge
it was not correct.
As mentioned in this issue:
https://issues.apache.org/jira/browse/SPARK-35723
And in
https://issues.apache.org/jira/browse/SPARK-37358
Although the default of using limits same as request make some sense, in
some cases it is much wiser to have a larger limit.
Let's say many executors / drivers are sharing the same instance - if the
limit and the request are the same - you must allocate to each pod the **max**
memory it would use, causing a huge over allocation.
It is very typical to have a momentarily high memory usage, every user
should have the power to decide if it makes sense for him to have a large
limit. it has its risks, but in many use cases the cost saving might worth it.
For now in our env we are actually using (for almost a year) webhooks to
override this behavior, I think it should be allowed natively.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
