Alberto Bortolan created SPARK-39358:
----------------------------------------
Summary: Have the arguments passed via
spark.executor.extraJavaOptions placed before the classpath when composing the
executors commandline
Key: SPARK-39358
URL: https://issues.apache.org/jira/browse/SPARK-39358
Project: Spark
Issue Type: Wish
Components: Spark Core
Affects Versions: 3.2.1, 2.4.8
Reporter: Alberto Bortolan
When submitting a job it's possible to pass java options to be applied to the
executor processes via the configuration parameter
{{spark.executor.extraJavaOptions}} as in
{noformat}
--conf 'spark.driver.extraJavaOptions=-Dmycompany.app.name=some_name'{noformat}
The command line for the executors is composed as
{noformat}
java <$SPARK_CONF_DIF/java.opts options> -cp <classpath> -Xmx
<extra_java_options> ...{noformat}
Since the classpath can be particularly long, it would be helpful for the
purpose of visibility when systems are monitored with tools that show the
first part of the command line of a process (e.g nmon), to get the extra java
options moved before classpath, as in
{noformat}
java <$SPARK_CONF_DIR/java.opts options> <extra_java_options> -cp <classpath>
-Xmx ...{noformat}
The type of extra options that would be passed in this way would be application
and submission dependent and so it would not be appropriate or even possible to
place them inside {{{}$SPARK_CONF_DIR/java-opts{}}}.
---
For reference, the code that builds the command line is
[buildSparkSubmitCommand|https://github.com/apache/spark/blob/4f25b3f71238a00508a356591553f2dfa89f8290/launcher/src/main/java/org/apache/spark/launcher/SparkSubmitCommandBuilder.java#L263],
where, one of the first actions is calling
[buildJavaCommand|https://github.com/apache/spark/blob/4f25b3f71238a00508a356591553f2dfa89f8290/launcher/src/main/java/org/apache/spark/launcher/AbstractCommandBuilder.java#L92]
that prepare the first part of the string from the "java" command to "-cp
classpath" included.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
