dongjoon-hyun commented on PR #48755:
URL: https://github.com/apache/spark/pull/48755#issuecomment-2457462259

   > I believe most of the other options' precedences are OK, which follow the 
`setIfMissing` logic. 
   
   It's opposite, isn't it. Apache Spark's configuration follows Java `Map` 
semantics instead of `setIfMessing`. For example, if you have `-c 
spark.app.name=A -c spark.app.name=B` in the same configuration. `B` is 
applied. 
   
   This becomes more serious when a Spark job has `-D` setting. For example, 
spark.app.name=A` and `-D spark.app.name=B` from different configurations like 
`--driver-java-options` and `spark.driver.extraJavaOptions` and 
`spark.driver.defaultJavaOptions` and K8s `SPARK_JAVA_OPT_*` and  
`spark.kubernetes.driverEnv.[EnvironmentVariableName]`.
   
   To me, it will cause more headache due to those corner cases, @yaooqinn .


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to