[
https://issues.apache.org/jira/browse/SPARK-21023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16045452#comment-16045452
]
Lantao Jin commented on SPARK-21023:
------------------------------------
If current behavior should be keep. I can add an environment variable.E.g
{{SPARK_CONF_REPLACE_ALLOWED}} with default value "true" and set to
{{childEnv}} map in AbstractCommandBuilder.class.
{code}
static final String SPARK_CONF_REPLACE_ALLOWED = "SPARK_CONF_REPLACE_ALLOWED";
{code}
Then we can export SPARK_CONF_REPLACE_ALLOWED=false in {{spark-env.sh}} to fix
this case and keep current behavior by default. Generally, the file
{{spark-env.sh}} deployed by infra team and protect by linux file permission
mechanism.
Of course, user can export to any value before submitting. But it means the
user definitely know what they want instead of the current unexpected result.
> Ignore to load default properties file is not a good choice from the
> perspective of system
> ------------------------------------------------------------------------------------------
>
> Key: SPARK-21023
> URL: https://issues.apache.org/jira/browse/SPARK-21023
> Project: Spark
> Issue Type: Improvement
> Components: Spark Submit
> Affects Versions: 2.1.1
> Reporter: Lantao Jin
> Priority: Minor
>
> The default properties file {{spark-defaults.conf}} shouldn't be ignore to
> load even though the submit arg {{--properties-file}} is set. The reasons are
> very easy to see:
> * Infrastructure team need continually update the {{spark-defaults.conf}}
> when they want set something as default for entire cluster as a tuning
> purpose.
> * Application developer only want to override the parameters they really want
> rather than others they even doesn't know (Set by infrastructure team).
> * The purpose of using {{\-\-properties-file}} from most of application
> developers is to avoid setting dozens of {{--conf k=v}}. But if
> {{spark-defaults.conf}} is ignored, the behaviour becomes unexpected finally.
> For example:
> Current implement
> ||Property name||Value in default||Value in user-special||Finally value||
> |spark.A|"foo"|"bar"|"bar"|
> |spark.B|"foo"|N/A|N/A|
> |spark.C|N/A|"bar"|"bar"|
> |spark.D|"foo"|"foo"|"foo"|
> |spark.E|"foo"|N/A|N/A|
> |spark.F|"foo"|N/A|N/A|
> Expected right implement
> ||Property name||Value in default||Value in user-special||Finally value||
> |spark.A|"foo"|"bar"|"bar"|
> |spark.B|"foo"|N/A|"foo"|
> |spark.C|N/A|"bar"|"bar"|
> |spark.D|"foo"|"foo"|"foo"|
> |spark.E|"foo"|"foo"|"foo"|
> |spark.F|"foo"|"foo"|"foo"|
> I can offer a patch to fix it if you think it make sense.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
