[
https://issues.apache.org/jira/browse/SPARK-21048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16388045#comment-16388045
]
Yaqub Alwan commented on SPARK-21048:
-------------------------------------
Commenting here because SPARK-21023 was closed and contrary to the remark "I
think this is confusing relative to any value it adds" I also have found this
behaviour of completely ignoring the spark-defaults.conf to be counterintuitive
(if I am being honest I would actually say I find it obnoxious, as they're
hardly defaults if they get ignored when not explicitly overridden) when
supplied with a properties file, and I would also like to see _some_ solution
to this problem, as a application developer doesn't want or need to know about
cluster level configuration in order to just set some application level
properties.
I would prefer to see something like --merge-properties-with-defaults in
combination with --properties-file but any implementation works. I am a little
concerned seeing the resistance to having this implemented, when the current
behaviour is not intuitive.
> Add an option --merged-properties-file to distinguish the configuration
> loading behavior
> ----------------------------------------------------------------------------------------
>
> Key: SPARK-21048
> URL: https://issues.apache.org/jira/browse/SPARK-21048
> Project: Spark
> Issue Type: Improvement
> Components: Spark Submit
> Affects Versions: 2.1.1
> Reporter: Lantao Jin
> Priority: Minor
>
> The problem description is the same as
> [SPARK-21023|https://issues.apache.org/jira/browse/SPARK-21023]. But
> different with that ticket. The purpose is not making sure the default
> properties file always be loaded. Instead, just offering other option to let
> user choose what they want.
> {quote}
> {{\-\-properties-file}} user-specified properties file which will replace the
> default properties file. deprecated.
> {{\-\-replaced-properties-file}} new option which equals the
> {{\-\-properties-file}} but more friendly.
> {{\-\-merged-properties-file}} user-specified properties file which will
> merge with the default properties file.
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
