shanyu zhao created SPARK-31028:
-----------------------------------
Summary: Add "-XX:ActiveProcessorCount" to Spark driver and
executor in Yarn mode
Key: SPARK-31028
URL: https://issues.apache.org/jira/browse/SPARK-31028
Project: Spark
Issue Type: Bug
Components: YARN
Affects Versions: 2.4.5
Reporter: shanyu zhao
When starting Spark driver and executors on Yarn cluster, the JVM process can
discover all CPU cores on the system and set thread-pool or GC threads based on
that value. We should limit what the JVM sees for the number of cores set by
the user (spark.driver.cores or spark.executor.cores) by
"-XX:ActiveProcessorCount", which was introduced in Java 8u191.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
