[ https://issues.apache.org/jira/browse/SPARK-2051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Guoqiang Li updated SPARK-2051: ------------------------------- Description: Spark configuration {{conf/spark-defaults.conf}}: {quote} spark.yarn.dist.archives /toona/conf spark.executor.extraClassPath ./conf spark.driver.extraClassPath ./conf {quote} ---- HDFS directory {{hadoop dfs -cat /toona/conf/toona.conf}} : {quote} redis.num=4 {quote} ---- The following command execution fails {code} YARN_CONF_DIR=/etc/hadoop/conf ./bin/spark-submit --num-executors 2 --driver-memory 2g --executor-memory 2g --master yarn-cluster --class toona.DeployTest toona-assembly.jar {code} ---- The following is the test code {code} package toona import com.typesafe.config.Config import com.typesafe.config.ConfigFactory object DeployTest { def main(args: Array[String]) { val conf = ConfigFactory.load("toona.conf") val redisNum = conf.getInt("redis.num") // Here will throw an `ConfigException` exception assert(redisNum == 4) } } {code} was: Spark configuration {{conf/spark-defaults.conf}}: {quote} spark.yarn.dist.archives /toona/conf spark.executor.extraClassPath ./conf spark.driver.extraClassPath ./conf {quote} ---- HDFS directory {{hadoop dfs -cat /toona/conf/toona.conf}} : {quote} redis.num=4 {quote} ---- The following command execution fails {code} YARN_CONF_DIR=/etc/hadoop/conf ./bin/spark-submit --num-executors 2 --driver-memory 2g --executor-memory 2g --master yarn-cluster --class toona.DeployTest toona-assembly.jar {code} ---- The following is testing the code {code} package toona import com.typesafe.config.Config import com.typesafe.config.ConfigFactory object DeployTest { def main(args: Array[String]) { val conf = ConfigFactory.load("toona.conf") val redisNum = conf.getInt("redis.num") // Here will throw an `ConfigException` exception assert(redisNum == 4) } } {code} > In yarn.ClientBase spark.yarn.dist.* do not work > ------------------------------------------------ > > Key: SPARK-2051 > URL: https://issues.apache.org/jira/browse/SPARK-2051 > Project: Spark > Issue Type: Bug > Components: YARN > Reporter: Guoqiang Li > > Spark configuration > {{conf/spark-defaults.conf}}: > {quote} > spark.yarn.dist.archives /toona/conf > spark.executor.extraClassPath ./conf > spark.driver.extraClassPath ./conf > {quote} > ---- > HDFS directory > {{hadoop dfs -cat /toona/conf/toona.conf}} : > {quote} > redis.num=4 > {quote} > ---- > The following command execution fails > {code} > YARN_CONF_DIR=/etc/hadoop/conf ./bin/spark-submit --num-executors 2 > --driver-memory 2g --executor-memory 2g --master yarn-cluster --class > toona.DeployTest toona-assembly.jar > {code} > ---- > The following is the test code > {code} > package toona > import com.typesafe.config.Config > import com.typesafe.config.ConfigFactory > object DeployTest { > def main(args: Array[String]) { > val conf = ConfigFactory.load("toona.conf") > val redisNum = conf.getInt("redis.num") // Here will throw an > `ConfigException` exception > assert(redisNum == 4) > } > } > {code} -- This message was sent by Atlassian JIRA (v6.2#6252)