Re: Shuffle files lifecycle

Thanks Silvio. On Mon, Jun 29, 2015 at 7:41 PM, Silvio Fiorito < silvio.fior...@granturing.com> wrote: > Regarding 1 and 2, yes shuffle output is stored on the worker local > disks and will be reused across jobs as long as they’re available. You can > identify when they’re used by seeing skipp

Re: Shuffle files lifecycle

Regarding 1 and 2, yes shuffle output is stored on the worker local disks and will be reused across jobs as long as they’re available. You can identify when they’re used by seeing skipped stages in the job UI. They are periodically cleaned up based on available space of the configured spark.loca

Re: Shuffle files lifecycle

Ah, for #3, maybe this is what *rdd.checkpoint *does! https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.rdd.RDD Thomas On Mon, Jun 29, 2015 at 7:12 PM, Thomas Gerber wrote: > Hello, > > It is my understanding that shuffle are written on disk and that they act > as chec

Re: Shuffle files not cleaned up (Spark 1.2.1)

Hi TD, That little experiment helped a bit. This time we did not see any exceptions for about 16 hours but eventually it did throw the same exceptions as before. The cleaning of the shuffle files also stopped much before these exceptions happened - about 7-1/2 hours after startup. I am not quite

Re: Shuffle files not cleaned up (Spark 1.2.1)

Hi TD, That may very well have been the case. There may be some delay on our output side. I have made a change just for testing that sends the output nowhere. I will see if that helps get rid of these errors. Then we can try to find out how we can optimize so that we do not lag. Questions: How ca

Re: Shuffle files not cleaned up (Spark 1.2.1)

What was the state of your streaming application? Was it falling behind with a large increasing scheduling delay? TD On Thu, Apr 23, 2015 at 11:31 AM, N B wrote: > Thanks for the response, Conor. I tried with those settings and for a > while it seemed like it was cleaning up shuffle files after

Re: Shuffle files not cleaned up (Spark 1.2.1)

Thanks for the response, Conor. I tried with those settings and for a while it seemed like it was cleaning up shuffle files after itself. However, after exactly 5 hours later it started throwing exceptions and eventually stopped working again. A sample stack trace is below. What is curious about 5

Re: Shuffle files not cleaned up (Spark 1.2.1)

Hi, We set the spark.cleaner.ttl to some reasonable time and also set spark.streaming.unpersist=true. Those together cleaned up the shuffle files for us. -Conor On Tue, Apr 21, 2015 at 8:18 AM, N B wrote: > We already do have a cron job in place to clean just the shuffle files. > However,

Re: Shuffle files not cleaned up (Spark 1.2.1)

We already do have a cron job in place to clean just the shuffle files. However, what I would really like to know is whether there is a "proper" way of telling spark to clean up these files once its done with them? Thanks NB On Mon, Apr 20, 2015 at 10:47 AM, Jeetendra Gangele wrote: > Write a

Re: Shuffle files not cleaned up (Spark 1.2.1)

Write a crone job for this like below 12 * * * * find $SPARK_HOME/work -cmin +1440 -prune -exec rm -rf {} \+ 32 * * * * find /tmp -type d -cmin +1440 -name "spark-*-*-*" -prune -exec rm -rf {} \+ 52 * * * * find $SPARK_LOCAL_DIR -mindepth 1 -maxdepth 1 -type d -cmin +1440 -name "spark-*-*-*" -p

RE: Shuffle files

Cc: Sunny Khatri; Lisonbee, Todd; u...@spark.incubator.apache.org Subject: Re: Shuffle files My observation is opposite. When my job runs under default spark.shuffle.manager, I don't see this exception. However, when it runs with SORT based, I start seeing this error? How would that be pos

Re: Shuffle files

1560.n3.nabble.com/quot-Too-many-open-files-quot-exception-on-reduceByKey-td2462.html >>> >>> Thanks, >>> >>> Todd >>> >>> -Original Message- >>> From: SK [mailto:skrishna...@gmail.com] >>> Sent: Tuesday, October 7, 2014 2:12

Re: Shuffle files

t;> Thanks, >> >> Todd >> >> -Original Message- >> From: SK [mailto:skrishna...@gmail.com] >> Sent: Tuesday, October 7, 2014 2:12 PM >> To: u...@spark.incubator.apache.org >> Subject: Re: Shuffle files >> >> - We set ulimit to 50. But I

Re: Shuffle files

files-quot-exception-on-reduceByKey-td2462.html > > Thanks, > > Todd > > -Original Message- > From: SK [mailto:skrishna...@gmail.com] > Sent: Tuesday, October 7, 2014 2:12 PM > To: u...@spark.incubator.apache.org > Subject: Re: Shuffle files > > - We set ulimit to

RE: Shuffle files

es-quot-exception-on-reduceByKey-td2462.html Thanks, Todd -Original Message- From: SK [mailto:skrishna...@gmail.com] Sent: Tuesday, October 7, 2014 2:12 PM To: u...@spark.incubator.apache.org Subject: Re: Shuffle files - We set ulimit to 50. But I still get the same "too many o

Re: Shuffle files

- We set ulimit to 50. But I still get the same "too many open files" warning. - I tried setting consolidateFiles to True, but that did not help either. I am using a Mesos cluster. Does Mesos have any limit on the number of open files? thanks -- View this message in context: http:/

Re: Shuffle files

Hi SK, For the problem with lots of shuffle files and the "too many open files" exception there are a couple options: 1. The linux kernel has a limit on the number of open files at once. This is set with ulimit -n, and can be set permanently in /etc/sysctl.conf or /etc/sysctl.d/. Try increasing

Re: Shuffle Files

>From BlockManager code + ShuffleMapTask code, it writes under spark.local.dir or java.io.tmpdir. val diskBlockManager = new DiskBlockManager(shuffleBlockManager, conf.get("spark.local.dir", System.getProperty("java.io.tmpdir"))) On Mon, Mar 3, 2014 at 10:45 PM, Usman Ghani wrote: > Whe