After setting SPARK_LOCAL_DIRS/SPARK_WORKER_DIR you need to restart your spark instances (stop-all.sh and start-all.sh), You can also try setting java.io.tmpdir while creating the SparkContext.
Thanks Best Regards On Wed, Mar 11, 2015 at 1:47 AM, Justin Yip <yipjus...@prediction.io> wrote: > Hello, > > I notice that when I run spark-submit, a temporary directory containing > all the jars and resource files is created under /tmp (for example, > /tmp/spark-fd1b77fc-50f4-4b1c-a122-5cf36969407c). > > Sometimes this directory gets cleanup after the job, but sometimes it > doesn't, which fills up my root directory. > > Is there a way to specify this directory so that it doesn't point to /tmp? > I tried SPARK_LOCAL_DIRS, but it doesn't help, neither does > SPARK_WORKER_DIR. > > Thanks. > > Justin >
