// assuming Spark 1.0 Hi Baoqiang,
In my experience for the standalone cluster you need to set SPARK_WORKER_DIR not SPARK_LOCAL_DIRS to control where shuffle files are written. I think this is a documentation issue that could be improved, as http://spark.apache.org/docs/latest/spark-standalone.html suggests using SPARK_LOCAL_DIRS for scratch, and I'm not sure that it actually does anything. Did you see anything in /mnt/data/tmp when you used SPARK_LOCAL_DIRS? Cheers! Andrew On Sat, Aug 9, 2014 at 7:21 AM, Baoqiang Cao <bqcaom...@gmail.com> wrote: > Hi > > I’m trying to using a specific dir for spark working directory since I > have limited space at /tmp. I tried: > 1) > export SPARK_LOCAL_DIRS=“/mnt/data/tmp” > or 2) > SPARK_LOCAL_DIRS=“/mnt/data/tmp” in spark-env.sh > > But neither worked, since the output of spark still saying > > ERROR DiskBlockObjectWriter: Uncaught exception while reverting partial > writes to file /tmp/spark-local-20140809134509-0502/34/shuffle_0_436_1 > java.io.FileNotFoundException: > /tmp/spark-local-20140809134509-0502/34/shuffle_0_436_1 (No space left on > device) > > anybody help with correctly setting up the “tmp” directory? > > Best, > Baoqiang Cao > Blog: http://baoqiang.org > Email: bqcaom...@gmail.com > > > > >
