[ https://issues.apache.org/jira/browse/SPARK-2051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Graves updated SPARK-2051: --------------------------------- Issue Type: Improvement (was: Bug) > spark.yarn.dist.* configs are not supported in yarn-cluster mode > ---------------------------------------------------------------- > > Key: SPARK-2051 > URL: https://issues.apache.org/jira/browse/SPARK-2051 > Project: Spark > Issue Type: Improvement > Components: YARN > Reporter: Guoqiang Li > Assignee: Guoqiang Li > > Spark configuration > {{conf/spark-defaults.conf}}: > {quote} > spark.yarn.dist.archives /toona/conf > spark.executor.extraClassPath ./conf > spark.driver.extraClassPath ./conf > {quote} > ---- > HDFS directory > {{hadoop dfs -cat /toona/conf/toona.conf}} : > {quote} > redis.num=4 > {quote} > ---- > The following command execution fails > {code} > YARN_CONF_DIR=/etc/hadoop/conf ./bin/spark-submit --num-executors 2 > --driver-memory 2g --executor-memory 2g --master yarn-cluster --class > toona.DeployTest toona-assembly.jar > {code} > ---- > The following is the test code > {code} > package toona > import com.typesafe.config.Config > import com.typesafe.config.ConfigFactory > object DeployTest { > def main(args: Array[String]) { > val conf = ConfigFactory.load("toona.conf") > val redisNum = conf.getInt("redis.num") // Here will throw an > `ConfigException` exception > assert(redisNum == 4) > } > } > {code} -- This message was sent by Atlassian JIRA (v6.2#6252)