Adam Binford created SPARK-49984:
------------------------------------
Summary: Duplicate JVM options
Key: SPARK-49984
URL: https://issues.apache.org/jira/browse/SPARK-49984
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 3.5.3
Reporter: Adam Binford
JVM options in spark.driver.defaultJavaOptions and
spark.executor.defaultJavaOptions end up being added multiple times to the
command line due to the way the Java module options and IPv6 options are added.
{{$ pyspark --conf spark.driver.defaultJavaOptions="-Dfoo=bar"}}
{{>>> spark.conf.get('spark.driver.extraJavaOptions')}}
{{'-Djava.net.preferIPv6Addresses=false -Dfoo=bar
-XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED
--add-opens=java.base/java.lang.invoke=ALL-UNNAMED
--add-opens=java.base/java.lang.reflect=ALL-UNNAMED
--add-opens=java.base/java.io=ALL-UNNAMED
--add-opens=java.base/java.net=ALL-UNNAMED
--add-opens=java.base/java.nio=ALL-UNNAMED
--add-opens=java.base/java.util=ALL-UNNAMED
--add-opens=java.base/java.util.concurrent=ALL-UNNAMED
--add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED
--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED
--add-opens=java.base/sun.nio.ch=ALL-UNNAMED
--add-opens=java.base/sun.nio.cs=ALL-UNNAMED
--add-opens=java.base/sun.security.action=ALL-UNNAMED
--add-opens=java.base/sun.util.calendar=ALL-UNNAMED
--add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED
-Djdk.reflect.useDirectMethodHandle=false -Dfoo=bar'}}
{{>>> spark.conf.get('spark.driver.defaultJavaOptions')}}
{{'-Dfoo=bar'}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
