[
https://issues.apache.org/jira/browse/SPARK-30949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-30949:
----------------------------------
Parent: SPARK-33005
Issue Type: Sub-task (was: Improvement)
> Driver cores in kubernetes are coupled with container resources, not
> spark.driver.cores
> ---------------------------------------------------------------------------------------
>
> Key: SPARK-30949
> URL: https://issues.apache.org/jira/browse/SPARK-30949
> Project: Spark
> Issue Type: Sub-task
> Components: Kubernetes, Spark Core
> Affects Versions: 3.1.0
> Reporter: Onur Satici
> Assignee: Onur Satici
> Priority: Major
> Fix For: 3.1.0
>
>
> Drivers submitted in kubernetes cluster mode set the parallelism of various
> components like 'RpcEnv', 'MemoryManager', 'BlockManager' from inferring the
> number of available cores by calling:
> {code:java}
> Runtime.getRuntime().availableProcessors()
> {code}
> By using this, spark applications running on java 8 or older incorrectly get
> the total number of cores in the host, ignoring the cgroup limits set by
> kubernetes (https://bugs.openjdk.java.net/browse/JDK-6515172). Java 9 and
> newer runtimes do not have this problem.
> Orthogonal to this, it is currently not possible to decouple resource limits
> on the driver container with the amount of parallelism of the various network
> and memory components listed above.
> My proposal is to use the 'spark.driver.cores' configuration to get the
> amount of parallelism, like we do for YARN
> (https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/SparkContext.scala#L2762-L2767).
> This will enable users to specify 'spark.driver.cores' to set parallelism,
> and specify 'spark.kubernetes.driver.requests.cores' to limit the resource
> requests of the driver container. Further, this will remove the need to call
> 'availableProcessors()', thus the same number of cores will be used for
> parallelism independent of the java runtime version.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
