You can set an arbitrary properties file by adding --properties-file argument to spark-submit. It would be nice to have spark-submit also look in SPARK_CONF_DIR as well by default. If you opened a JIRA for that I'm sure someone would pick it up.
On Tue, Jun 3, 2014 at 7:47 AM, Eugen Cepoi <cepoi.eu...@gmail.com> wrote: > Is it on purpose that when setting SPARK_CONF_DIR spark submit still loads > the properties file from SPARK_HOME/conf/spark-defauls.conf ? > > IMO it would be more natural to override what is defined in SPARK_HOME/conf > by SPARK_CONF_DIR when defined (and SPARK_CONF_DIR being overriden by > command line args). > > Eugen
