pan3793 commented on PR #45504:
URL: https://github.com/apache/spark/pull/45504#issuecomment-2665957194
@dongjoon-hyun @mridulm @robreeves, the added configuration is tricky, for
example,
```
$ bin/spark-shell --conf
spark.executor.extraJavaOptions="-Dspark.shutdown.timeout=60s"
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 4.0.0-preview2
/_/
Using Scala version 2.13.14 (OpenJDK 64-Bit Server VM, Java 17.0.13)
Type in expressions to have them evaluated.
Type :help for more information.
25/02/18 22:54:42 WARN NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
25/02/18 22:54:43 ERROR SparkContext: Error initializing SparkContext.
java.lang.Exception: spark.executor.extraJavaOptions is not allowed to set
Spark options (was '-Dspark.shutdown.timeout=60s'). Set them directly on a
SparkConf or in a properties file when using ./bin/spark-submit.
at
org.apache.spark.SparkConf.$anonfun$validateSettings$5(SparkConf.scala:527)
at
org.apache.spark.SparkConf.$anonfun$validateSettings$5$adapted(SparkConf.scala:522)
...
```
instead, we can use `--conf spark.hadoop.hadoop.service.shutdown.timeout=60`
to override Hadoop configuration by Spark conf without any changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
