My ulimit is already "unlimited". Wouldn't setting it to 18000 be decreasing it? Could it be an issue other than ulimit? Because I can't increase it beyond "unlimited".
Thanks, Mike From: Mayur Rustagi <mayur.rust...@gmail.com<mailto:mayur.rust...@gmail.com>> Reply-To: "user@spark.incubator.apache.org<mailto:user@spark.incubator.apache.org>" <user@spark.incubator.apache.org<mailto:user@spark.incubator.apache.org>> Date: Thursday, February 13, 2014 12:58 PM To: "user@spark.incubator.apache.org<mailto:user@spark.incubator.apache.org>" <user@spark.incubator.apache.org<mailto:user@spark.incubator.apache.org>> Subject: Re: [External] Re: Too many open files Easiest is ulimit -n 18000 to your conf/spark-env.sh Restart the cluster.. make sure you change the limits at OS level as well. Regards Mayur Mayur Rustagi Ph: +919632149971 h<https://twitter.com/mayur_rustagi>ttp://www.sigmoidanalytics.com<http://www.sigmoidanalytics.com> https://twitter.com/mayur_rustagi On Thu, Feb 13, 2014 at 9:51 AM, Korb, Michael [USA] <korb_mich...@bah.com<mailto:korb_mich...@bah.com>> wrote: No I don't. I ran all Spark processes as a user with ulimit = unlimited. From: Mayur Rustagi <mayur.rust...@gmail.com<mailto:mayur.rust...@gmail.com>> Reply-To: "user@spark.incubator.apache.org<mailto:user@spark.incubator.apache.org>" <user@spark.incubator.apache.org<mailto:user@spark.incubator.apache.org>> Date: Thursday, February 13, 2014 12:34 PM To: "user@spark.incubator.apache.org<mailto:user@spark.incubator.apache.org>" <user@spark.incubator.apache.org<mailto:user@spark.incubator.apache.org>> Subject: [External] Re: Too many open files The limit could be on any of the machines(including the master). Do you have ganglia setup? Mayur Rustagi Ph: +919632149971<tel:%2B919632149971> h<https://twitter.com/mayur_rustagi>ttp://www.sigmoidanalytics.com<http://www.sigmoidanalytics.com> https://twitter.com/mayur_rustagi On Thu, Feb 13, 2014 at 7:13 AM, Korb, Michael [USA] <korb_mich...@bah.com<mailto:korb_mich...@bah.com>> wrote: Hi, When I submit a job to a cluster, I get a large string of errors like this: WARN TaskSetManager: Loss was due to java.io.FileNotFoundException java.io.FileNotFoundException: /tmp/spark-local* (Too many open files) All the answers I can find say to increase ulimit, but I have it set to unlimited (as the user running the spark daemons as well as the user submitting the job) and am still getting the error. I'm attempting to create an RDD like this: sc.textFile(/path/to/files/*).persist(StorageLevel.MEMORY_ONLY_SER()), and run a series of maps and filters on the data. There are about 2k files for a total of about 230g of data, and my current cluster is 3 nodes, 32 cores each, with spark.executor.memory set to 32g. I've tried different StorageLevel settings but still have the same error. Interestingly, the job works if I write and submit with pyspark, but I want to get it working in Java. Thanks, Mike
