Hi, No, there is no way to change local dir paths after Worker initialized. That is, dir paths are cached when a first executor is launched, then following executors reference the paths. Details can be found in codes below; https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala#L449 https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/util/Utils.scala#L679
On Wed, Feb 24, 2016 at 5:13 PM, Jung <jb_j...@naver.com> wrote: > Hi all, > In the standalone mode, spark.local.dir is ignored after Spark Worker > launched. > This is a scenario assuming that we have 1 master node and 1 worker node. > 1. $SPARK_HOME/sbin/start-all.sh to launch Spark Master&Worker > 2. Modify worker node configuration($SPARK_HOME/conf/spark-defaults.conf) > spark.local.dir to another directory( /tmp_new). > 3. run spark-shell from master node. > > Executor in this scenario creates scratch directory like > "spark-bb0876f2-7fa9-4f15-b790-24252183a4f1" under /tmp not /tmp_new. > Because worker set immutable SparkConf instance at the first time it > launched and refer to this variable when create new executor which wants to > change its scratch dir. > Can I change application's spark.local.dir without restarting spark > workers? > > Thanks, > Jung -- --- Takeshi Yamamuro
