[ https://issues.apache.org/jira/browse/SPARK-21048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16388045#comment-16388045 ]
Yaqub Alwan commented on SPARK-21048: ------------------------------------- Commenting here becauseĀ SPARK-21023 was closed and contrary to the remark "I think this is confusing relative to any value it adds" I also have found this behaviour of completely ignoring the spark-defaults.conf to be counterintuitive (if I am being honest I would actually say I find it obnoxious, as they're hardly defaults if they get ignored when not explicitly overridden) when supplied with a properties file, and I would also like to seeĀ _some_ solution to this problem, as a application developer doesn't want or need to know about cluster level configuration in order to just set some application level properties. I would prefer to see something like --merge-properties-with-defaults in combination with --properties-file but any implementation works. I am a little concerned seeing the resistance to having this implemented, when the current behaviour is not intuitive. > Add an option --merged-properties-file to distinguish the configuration > loading behavior > ---------------------------------------------------------------------------------------- > > Key: SPARK-21048 > URL: https://issues.apache.org/jira/browse/SPARK-21048 > Project: Spark > Issue Type: Improvement > Components: Spark Submit > Affects Versions: 2.1.1 > Reporter: Lantao Jin > Priority: Minor > > The problem description is the same as > [SPARK-21023|https://issues.apache.org/jira/browse/SPARK-21023]. But > different with that ticket. The purpose is not making sure the default > properties file always be loaded. Instead, just offering other option to let > user choose what they want. > {quote} > {{\-\-properties-file}} user-specified properties file which will replace the > default properties file. deprecated. > {{\-\-replaced-properties-file}} new option which equals the > {{\-\-properties-file}} but more friendly. > {{\-\-merged-properties-file}} user-specified properties file which will > merge with the default properties file. > {quote} -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org