Christian Thiel created SPARK-35723:
---------------------------------------
Summary: [K8s] Spark executors memory request should be allowed to
deviate from limit
Key: SPARK-35723
URL: https://issues.apache.org/jira/browse/SPARK-35723
Project: Spark
Issue Type: Improvement
Components: Kubernetes
Affects Versions: 3.1.2
Reporter: Christian Thiel
Currently the driver and executor memor requests always equals the limit.
As stated in [SPARK-23825|https://issues.apache.org/jira/browse/SPARK-23825],
this is a reasonable default and is especially important for the driver.
For executors however, it might be usefull for users to deviate from this
default for executors.
In typical development environments on K8s, the namespace quotas are an upper
bound to the memory request that is possible.
The limits however can be much higher. For development, spark is often run in
client mode. While the driver should request the memory it needs, we want to
leverage all the resources of the cluster whith executors if they are free -
and can live with an executor maybe beeing killed eventually.
Thus I propose the introduction of {{spark.{driver,executor}.limit.memory}}
similar to the {{spark.{driver,executor}.limit.cpu}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
