John Zhuge created SPARK-53209:
----------------------------------
Summary: Add "-XX:ActiveProcessorCount" to Spark driver and
executor in Yarn mode
Key: SPARK-53209
URL: https://issues.apache.org/jira/browse/SPARK-53209
Project: Spark
Issue Type: Bug
Components: Spark Core, YARN
Affects Versions: 2.4.5
Reporter: John Zhuge
When starting Spark driver and executors on Yarn cluster, the JVM process can
discover all CPU cores on the system and set thread-pool or GC threads based on
that value. We should limit what the JVM sees for the number of cores set by
the user (spark.driver.cores or spark.executor.cores) by
"-XX:ActiveProcessorCount", which was introduced in Java 8u191.
Especially in running Spark on Yarn inside Kubernetes container, the number of
CPU cores discovered sometimes is 1, which means it always use 1 thread in the
default thread pool, or GC threads.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
