Yes, you can set the SPARK_LOCAL_DIR in the spark-env.sh or spark.local.dir in the spark-defaults.conf file, then it would use this location for the shuffle writes etc.
Thanks Best Regards On Wed, Aug 26, 2015 at 6:56 PM, <michael.engl...@nomura.com> wrote: > Hi, > > > > I am having issues with /tmp space filling up during Spark jobs because > Spark-on-YARN uses the yarn.nodemanager.local-dirs for shuffle space. I > noticed this message appears when submitting Spark-on-YARN jobs: > > > > WARN SparkConf: In Spark 1.0 and later spark.local.dir will be overridden > by the value set by the cluster manager (via SPARK_LOCAL_DIRS in > mesos/standalone and LOCAL_DIRS in YARN). > > > > I can’t find much documentation on where to set the LOCAL_DIRS property. > Please can someone advise whether this is a yarn-env.sh or a spark-env.sh > property and whether it would then use the directory specified by this env > variable as a shuffle area instead of the default > yarn.nodemanager.local-dirs location? > > > > Thanks, > > Mike > > This e-mail (including any attachments) is private and confidential, may > contain proprietary or privileged information and is intended for the named > recipient(s) only. Unintended recipients are strictly prohibited from > taking action on the basis of information in this e-mail and must contact > the sender immediately, delete this e-mail (and all attachments) and > destroy any hard copies. Nomura will not accept responsibility or liability > for the accuracy or completeness of, or the presence of any virus or > disabling code in, this e-mail. If verification is sought please request a > hard copy. Any reference to the terms of executed transactions should be > treated as preliminary only and subject to formal written confirmation by > Nomura. Nomura reserves the right to retain, monitor and intercept e-mail > communications through its networks (subject to and in accordance with > applicable laws). No confidentiality or privilege is waived or lost by > Nomura by any mistransmission of this e-mail. Any reference to "Nomura" is > a reference to any entity in the Nomura Holdings, Inc. group. Please read > our Electronic Communications Legal Notice which forms part of this e-mail: > http://www.Nomura.com/email_disclaimer.htm >
