Can you try setting SPARK_LOCAL_DIRS in spark-env.sh ? Cheers
On Fri, Feb 6, 2015 at 7:30 AM, Joe Wass <jw...@crossref.org> wrote: > I'm running on EC2 and I want to set the directory to use on the slaves > (mounted EBS volumes). > > I have set: > spark.local.dir /vol3/my-spark-dir > in > /root/spark/conf/spark-defaults.conf > > and replicated to all nodes. I have verified that in the console the value > in the config corresponds. I have checked that these values are present in > nodes. > > But it's still creating temp files in the wrong (default) place: > > /mnt2/spark > > How do I get my slaves to pick up this value? How can I verify that they > have? > > Thanks! > > Joe >
