[ https://issues.apache.org/jira/browse/SPARK-21023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-21023: ------------------------------------ Assignee: (was: Apache Spark) > Ignore to load default properties file is not a good choice from the > perspective of system > ------------------------------------------------------------------------------------------ > > Key: SPARK-21023 > URL: https://issues.apache.org/jira/browse/SPARK-21023 > Project: Spark > Issue Type: Improvement > Components: Spark Submit > Affects Versions: 2.1.1 > Reporter: Lantao Jin > Priority: Minor > > The default properties file {{spark-defaults.conf}} shouldn't be ignore to > load even though the submit arg {{--properties-file}} is set. The reasons are > very easy to see: > * Infrastructure team need continually update the {{spark-defaults.conf}} > when they want set something as default for entire cluster as a tuning > purpose. > * Application developer only want to override the parameters they really want > rather than others they even doesn't know (Set by infrastructure team). > * The purpose of using {{\-\-properties-file}} from most of application > developers is to avoid setting dozens of {{--conf k=v}}. But if > {{spark-defaults.conf}} is ignored, the behaviour becomes unexpected finally. > All this caused by below codes: > {code} > private Properties loadPropertiesFile() throws IOException { > Properties props = new Properties(); > File propsFile; > if (propertiesFile != null) { > // default conf property file will not be loaded when app developer use > --properties-file as a submit args > propsFile = new File(propertiesFile); > checkArgument(propsFile.isFile(), "Invalid properties file '%s'.", > propertiesFile); > } else { > propsFile = new File(getConfDir(), DEFAULT_PROPERTIES_FILE); > } > //... > return props; > } > {code} > I can offer a patch to fix it if you think it make sense. -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org