[jira] [Commented] (SPARK-21048) Add an option --merged-properties-file to distinguish the configuration loading behavior

2018-03-06 Thread Yaqub Alwan (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-21048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388045#comment-16388045
 ] 

Yaqub Alwan commented on SPARK-21048:
-

Commenting here because SPARK-21023 was closed and contrary to the remark "I 
think this is confusing relative to any value it adds" I also have found this 
behaviour of completely ignoring the spark-defaults.conf to be counterintuitive 
(if I am being honest I would actually say I find it obnoxious, as they're 
hardly defaults if they get ignored when not explicitly overridden) when 
supplied with a properties file, and I would also like to see _some_ solution 
to this problem, as a application developer doesn't want or need to know about 
cluster level configuration in order to just set some application level 
properties.

I would prefer to see something like --merge-properties-with-defaults in 
combination with --properties-file but any implementation works. I am a little 
concerned seeing the resistance to having this implemented, when the current 
behaviour is not intuitive.

> Add an option --merged-properties-file to distinguish the configuration 
> loading behavior
> 
>
> Key: SPARK-21048
> URL: https://issues.apache.org/jira/browse/SPARK-21048
> Project: Spark
>  Issue Type: Improvement
>  Components: Spark Submit
>Affects Versions: 2.1.1
>Reporter: Lantao Jin
>Priority: Minor
>
> The problem description is the same as 
> [SPARK-21023|https://issues.apache.org/jira/browse/SPARK-21023]. But 
> different with that ticket. The purpose is not making sure the default 
> properties file always be loaded. Instead, just offering other option to let 
> user choose what they want.
> {quote}
> {{\-\-properties-file}} user-specified properties file which will replace the 
> default properties file. deprecated.
> {{\-\-replaced-properties-file}} new option which equals the 
> {{\-\-properties-file}} but more friendly. 
> {{\-\-merged-properties-file}} user-specified properties file which will 
> merge with the default properties file.
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-21048) Add an option --merged-properties-file to distinguish the configuration loading behavior

2017-06-10 Thread Lantao Jin (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-21048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16045539#comment-16045539
 ] 

Lantao Jin commented on SPARK-21048:


Thank you for pointing out about JIRA. Confusing name can be explain by 
document. Better than miss default configuration when use \-\-properties-file 
option. That the key point I try to fix. Any idea? 

> Add an option --merged-properties-file to distinguish the configuration 
> loading behavior
> 
>
> Key: SPARK-21048
> URL: https://issues.apache.org/jira/browse/SPARK-21048
> Project: Spark
>  Issue Type: Improvement
>  Components: Spark Submit
>Affects Versions: 2.1.1
>Reporter: Lantao Jin
>Priority: Minor
>
> The problem description is the same as 
> [SPARK-21023|https://issues.apache.org/jira/browse/SPARK-21023]. But 
> different with that ticket. The purpose is not making sure the default 
> properties file always be loaded. Instead, just offering other option to let 
> user choose what they want.
> {quote}
> {{\-\-properties-file}} user-specified properties file which will replace the 
> default properties file. deprecated.
> {{\-\-replaced-properties-file}} new option which equals the 
> {{\-\-properties-file}} but more friendly. 
> {{\-\-merged-properties-file}} user-specified properties file which will 
> merge with the default properties file.
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-21048) Add an option --merged-properties-file to distinguish the configuration loading behavior

2017-06-10 Thread Sean Owen (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-21048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16045533#comment-16045533
 ] 

Sean Owen commented on SPARK-21048:
---

I think this is confusing relative to any value it adds, and don't think this 
should be done.
(You shouldn't open new JIRAs for the same issue as it forks the discussion)

> Add an option --merged-properties-file to distinguish the configuration 
> loading behavior
> 
>
> Key: SPARK-21048
> URL: https://issues.apache.org/jira/browse/SPARK-21048
> Project: Spark
>  Issue Type: Improvement
>  Components: Spark Submit
>Affects Versions: 2.1.1
>Reporter: Lantao Jin
>Priority: Minor
>
> The problem description is the same as 
> [SPARK-21023|https://issues.apache.org/jira/browse/SPARK-21023]. But 
> different with that ticket. The purpose is not making sure the default 
> properties file always be loaded. Instead, just offering other option to let 
> user choose what they want.
> {quote}
> {{\-\-properties-file}} user-specified properties file which will replace the 
> default properties file. deprecated.
> {{\-\-replaced-properties-file}} new option which equals the 
> {{\-\-properties-file}} but more friendly. 
> {{\-\-merged-properties-file}} user-specified properties file which will 
> merge with the default properties file.
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-21048) Add an option --merged-properties-file to distinguish the configuration loading behavior

2017-06-10 Thread Lantao Jin (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-21048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16045502#comment-16045502
 ] 

Lantao Jin commented on SPARK-21048:


Will push a PR soon after a short discussing.

> Add an option --merged-properties-file to distinguish the configuration 
> loading behavior
> 
>
> Key: SPARK-21048
> URL: https://issues.apache.org/jira/browse/SPARK-21048
> Project: Spark
>  Issue Type: Improvement
>  Components: Spark Submit
>Affects Versions: 2.1.1
>Reporter: Lantao Jin
>Priority: Minor
>
> The problem description is the same as 
> [SPARK-21023|https://issues.apache.org/jira/browse/SPARK-21023]. But 
> different with that ticket. The purpose is not making sure the default 
> properties file always be loaded. Instead, just offering other option to let 
> user choose what they want.
> {quote}
> {{\-\-properties-file}} user-specified properties file which will replace the 
> default properties file. deprecated.
> {{\-\-replaced-properties-file}} new option which equals the 
> {{\-\-properties-file}} but more friendly. 
> {{\-\-merged-properties-file}} user-specified properties file which will 
> merge with the default properties file.
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org