Nicolas Fraison created SPARK-44242:
---------------------------------------
Summary: Spark job submission failed because Xmx string is
available on one parameter provided into spark.driver.extraJavaOptions
Key: SPARK-44242
URL: https://issues.apache.org/jira/browse/SPARK-44242
Project: Spark
Issue Type: Bug
Components: Spark Submit
Affects Versions: 3.4.1, 3.3.2
Reporter: Nicolas Fraison
The spark-submit command failed if Xmx string is found on any parameters
provided to spark.driver.extraJavaOptions.
For ex. running this spark-submit command line
{code:java}
./bin/spark-submit --class org.apache.spark.examples.SparkPi --conf
"spark.driver.extraJavaOptions=-Dtest=Xmx"
examples/jars/spark-examples_2.12-3.4.1.jar 100{code}
failed due to
{code:java}
Error: Not allowed to specify max heap(Xmx) memory settings through java
options (was -Dtest=Xmx). Use the corresponding --driver-memory or
spark.driver.memory configuration instead.{code}
The check performed in
[https://github.com/apache/spark/blob/master/launcher/src/main/java/org/apache/spark/launcher/SparkSubmitCommandBuilder.java#L314]
seems to broad
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
