[
https://issues.apache.org/jira/browse/SPARK-43496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17750791#comment-17750791
]
Laurenceau Julien commented on SPARK-43496:
-------------------------------------------
Hi,
I would like to suggest to go beyond that :
As is spark on kubernetes always sets the following attributes to executors and
driver pods:
{code:java}
requests:
cpu: "1"
memory: "1234M"
limits:
memory: "1234M" {code}
with the values coming from the _[driver|executor]-memory_ and
_[driver|executor]-cores_ options.
This is good, except for resource utilization !
On most dev environment it's would be ok no to set any resource requests and
limits. One could just rely on usual JVM options.
Besides it would be nice if these settings could be user defined at different
values than JVM values so that we would be able to manage overcommitment
through it.
Example:
* options : _[driver|executor]-memory_ and _[driver|executor]-cores_ are used
to set JVM options.
If no custom definition it also sets pods requests & limits to stay consistent
* new options:
_spark.kubernetes.driver.requests.cpu_
_spark.kubernetes.driver.requests.memory_
_spark.kubernetes.driver.limits.cpu_
_spark.kubernetes.driver.limits.memory_
_spark.kubernetes.executor.requests.cpu_
_spark.kubernetes.executor.requests.memory_
_spark.kubernetes.executor.limits.cpu_
_spark.kubernetes.executor.limits.memory_
if unset then stay consistent with current behavior
if set to 0 then disable this definition
This would also solve the issue that driver/executor core is defined as an
Integer and cannot be 0.5 for a driver.
> Have a separate config for Memory limits for kubernetes pods
> ------------------------------------------------------------
>
> Key: SPARK-43496
> URL: https://issues.apache.org/jira/browse/SPARK-43496
> Project: Spark
> Issue Type: Improvement
> Components: Kubernetes
> Affects Versions: 3.4.0
> Reporter: Alexander Yerenkow
> Priority: Major
>
> Whole allocated memory to JVM is set into pod resources as both request and
> limits.
> This means there's not a way to use more memory for burst-like jobs in a
> shared environment.
> For example, if spark job uses external process (outside of JVM) to access
> data, a bit of extra memory required for that, and having configured higher
> limits for mem could be of use.
> Another thought here - have a way to configure different JVM/ pod memory
> request also could be a valid use case.
>
> Github PR: [https://github.com/apache/spark/pull/41067]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
