That's a worker setting which cleans up the files left behind by executors, so spark.cleaner.ttl isn't at the RDD level. After https://issues.apache.org/jira/browse/SPARK-1860 the cleaner won't clean up directories left by running executors.
On Fri, Jan 9, 2015 at 7:38 AM, <michael.engl...@nomura.com> wrote: > Thanks, I imagine this will kill any cached RDDs if their files are > beyond the ttl? > > > > Thanks > > > > > > *From:* Raghavendra Pandey [mailto:raghavendra.pan...@gmail.com] > *Sent:* 09 January 2015 15:29 > *To:* England, Michael (IT/UK); user@spark.apache.org > *Subject:* Re: Cleaning up spark.local.dir automatically > > > > You may like to look at spark.cleaner.ttl configuration which is infinite > by default. Spark has that configuration to delete temp files time to time. > > On Fri Jan 09 2015 at 8:34:10 PM <michael.engl...@nomura.com> wrote: > > Hi, > > > > Is there a way of automatically cleaning up the spark.local.dir after a > job has been run? I have noticed a large number of temporary files have > been stored here and are not cleaned up. The only solution I can think of > is to run some sort of cron job to delete files older than a few days. I am > currently using a mixture of standalone and YARN spark builds. > > > > Thanks, > > Michael > > > > > > This e-mail (including any attachments) is private and confidential, may > contain proprietary or privileged information and is intended for the named > recipient(s) only. Unintended recipients are strictly prohibited from > taking action on the basis of information in this e-mail and must contact > the sender immediately, delete this e-mail (and all attachments) and > destroy any hard copies. Nomura will not accept responsibility or liability > for the accuracy or completeness of, or the presence of any virus or > disabling code in, this e-mail. If verification is sought please request a > hard copy. Any reference to the terms of executed transactions should be > treated as preliminary only and subject to formal written confirmation by > Nomura. Nomura reserves the right to retain, monitor and intercept e-mail > communications through its networks (subject to and in accordance with > applicable laws). No confidentiality or privilege is waived or lost by > Nomura by any mistransmission of this e-mail. Any reference to "Nomura" is > a reference to any entity in the Nomura Holdings, Inc. group. Please read > our Electronic Communications Legal Notice which forms part of this e-mail: > http://www.Nomura.com/email_disclaimer.htm > > This e-mail (including any attachments) is private and confidential, may > contain proprietary or privileged information and is intended for the named > recipient(s) only. Unintended recipients are strictly prohibited from > taking action on the basis of information in this e-mail and must contact > the sender immediately, delete this e-mail (and all attachments) and > destroy any hard copies. Nomura will not accept responsibility or liability > for the accuracy or completeness of, or the presence of any virus or > disabling code in, this e-mail. If verification is sought please request a > hard copy. Any reference to the terms of executed transactions should be > treated as preliminary only and subject to formal written confirmation by > Nomura. Nomura reserves the right to retain, monitor and intercept e-mail > communications through its networks (subject to and in accordance with > applicable laws). No confidentiality or privilege is waived or lost by > Nomura by any mistransmission of this e-mail. Any reference to "Nomura" is > a reference to any entity in the Nomura Holdings, Inc. group. Please read > our Electronic Communications Legal Notice which forms part of this e-mail: > http://www.Nomura.com/email_disclaimer.htm >
