Yaqub Alwan commented on SPARK-21048:

Commenting here becauseĀ SPARK-21023 was closed and contrary to the remark "I 
think this is confusing relative to any value it adds" I also have found this 
behaviour of completely ignoring the spark-defaults.conf to be counterintuitive 
(if I am being honest I would actually say I find it obnoxious, as they're 
hardly defaults if they get ignored when not explicitly overridden) when 
supplied with a properties file, and I would also like to seeĀ _some_ solution 
to this problem, as a application developer doesn't want or need to know about 
cluster level configuration in order to just set some application level 

I would prefer to see something like --merge-properties-with-defaults in 
combination with --properties-file but any implementation works. I am a little 
concerned seeing the resistance to having this implemented, when the current 
behaviour is not intuitive.

> Add an option --merged-properties-file to distinguish the configuration 
> loading behavior
> ----------------------------------------------------------------------------------------
>                 Key: SPARK-21048
>                 URL: https://issues.apache.org/jira/browse/SPARK-21048
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Submit
>    Affects Versions: 2.1.1
>            Reporter: Lantao Jin
>            Priority: Minor
> The problem description is the same as 
> [SPARK-21023|https://issues.apache.org/jira/browse/SPARK-21023]. But 
> different with that ticket. The purpose is not making sure the default 
> properties file always be loaded. Instead, just offering other option to let 
> user choose what they want.
> {quote}
> {{\-\-properties-file}} user-specified properties file which will replace the 
> default properties file. deprecated.
> {{\-\-replaced-properties-file}} new option which equals the 
> {{\-\-properties-file}} but more friendly. 
> {{\-\-merged-properties-file}} user-specified properties file which will 
> merge with the default properties file.
> {quote}

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to