Re: Setting Spark TMP Directory in Cluster Mode

Any takers? :-) --- Mufeed Usman My LinkedIn | My Social Cause | My Blogs : LiveJournal On Mon, Sep 28, 2015 at 10:19 AM, mufy wrote: > Hello Akhil, > > I do not see how tha

Re: Setting Spark TMP Directory in Cluster Mode

Hello Akhil, I do not see how that would work for a YARN cluster mode execution since the local directories used by the Spark executors and the Spark driver are the local directories that are configured for YARN (yarn.nodemanager.local-dirs). If you specify a different path with SPARK_LOCAL_DIRS,

Re: Setting Spark TMP Directory in Cluster Mode

Try with spark.local.dir in the spark-defaults.conf or SPARK_LOCAL_DIR in the spark-env.sh file. Thanks Best Regards On Fri, Sep 25, 2015 at 2:14 PM, mufy wrote: > Faced with an issue where Spark temp files get filled under > /opt/spark-1.2.1/tmp on the local filesystem on the worker nodes. Whi

Setting Spark TMP Directory in Cluster Mode

Faced with an issue where Spark temp files get filled under /opt/spark-1.2.1/tmp on the local filesystem on the worker nodes. Which parameter/configuration sets that location? The spark-env.sh file has the folders set as, export SPARK_HOME=/opt/spark-1.2.1 export SPARK_WORKER_DIR=$SPARK_HOME/tmp