[
https://issues.apache.org/jira/browse/SPARK-2664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14074108#comment-14074108
]
Patrick Wendell commented on SPARK-2664:
----------------------------------------
Hey Sandy,
The reason why we originally allowed spark-defaults.conf to support spark
options that have corresponding flags (which is 1 here) is because there was no
other way for users to set things like the master in a configuration file. This
seems like something we need to support (at least, IIRC it was one reason some
packagers wanted a configuration file in the first place). So my proposal here
was to just treat --conf as the same way. I'd also be okay to just throw an
exception if users try to set one of these as a --conf when it corresponds to a
flag, but then it deviates a bit from the behavior of the config file which
could be confusing.
In terms of adding new configs for flags that don't currently have a
corresponding conf. Personally, I'm also open to doing that. It would certainly
simplify the fact that we have two different concepts at the moment for which
there is not a 1:1 mapping. In my experience users have been most confused
about the fact that those flags and spark conf properties are
partially-but-not-completely overlapping. They haven't been confused about
precedence order as much, since we state it clearly.
In the shorter term, given that we can't revert the behavior of
spark-defaults.conf, I'd prefer to either use that same behavior for flags (the
original proposal) or to throw an exception if any of the "reserved" properties
are set via the --conf flag instead of their dedicated flag (we can always make
it accept more liberally later). Otherwise it's super confusing what happens if
the user sets --conf spark.master and also --master.
> Deal with `--conf` options in spark-submit that relate to flags
> ---------------------------------------------------------------
>
> Key: SPARK-2664
> URL: https://issues.apache.org/jira/browse/SPARK-2664
> Project: Spark
> Issue Type: Bug
> Reporter: Patrick Wendell
> Assignee: Sandy Ryza
> Priority: Blocker
>
> If someone sets a spark conf that relates to an existing flag `--master`, we
> should set it correctly like we do with the defaults file. Otherwise it can
> have confusing semantics. I noticed this after merging it, otherwise I would
> have mentioned it in the review.
> I think it's as simple as modifying loadDefaults to check the user-supplied
> options also. We might change it to loadUserProperties since it's no longer
> just the defaults file.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
