If you are using systemd, you will need to specify the limit in the service file. I had run into this problem and discovered the solution from the following references: * https://bugzilla.redhat.com/show_bug.cgi?id=754285#c1 * http://serverfault.com/a/678861
On Fri, Feb 5, 2016 at 1:18 PM, Nirav Patel <npa...@xactlycorp.com> wrote: > For centos there's also /etc/security/limits.d/90-nproc.conf that may > need modifications. > > Services that you expect to use new limits needs to be restarted. Simple > thing to do is to reboot the machine. > > On Fri, Feb 5, 2016 at 3:59 AM, Ted Yu <yuzhih...@gmail.com> wrote: > >> bq. and *"session required pam_limits.so"*. >> >> What was the second file you modified ? >> >> Did you make the change on all the nodes ? >> >> Please see the verification step in >> https://easyengine.io/tutorials/linux/increase-open-files-limit/ >> >> On Fri, Feb 5, 2016 at 1:42 AM, Mohamed Nadjib MAMI <m...@iai.uni-bonn.de >> > wrote: >> >>> Hello all, >>> >>> I'm getting the famous *java.io.FileNotFoundException: ... (Too many >>> open files) *exception. What seemed to have helped people out, it >>> haven't for me. I tried to set the ulimit via the command line *"ulimit >>> -n"*, then I tried to add the following lines to >>> *"/etc/security/limits.conf"* file: >>> >>> ** - nofile 1000000* >>> *root soft nofile 1000000* >>> *root hard nofile 1000000* >>> *hduser soft nofile 1000000* >>> *hduser hard nofile 1000000* >>> >>> ...then I added this line *"session required pam_limits.so"* to the two >>> files* "/etc/pam.d/common-session"* and *"session required >>> pam_limits.so"*. The I logged-out/logged-in. First, I tried only the >>> first line (** - nofile 1000000**)*, then added the 2nd and the 3rd >>> (root...), then added the last two lines (hduser...), no effect. Weirdly >>> enough, when I check with the command *"ulimit -n"* it returns the >>> correct value of 1000000. >>> >>> I then added *"ulimit -n 1000000"* to *"spark-env.sh"* in the master >>> and in each of my workers, no effect. >>> >>> What else could it be besides changing the ulimit setting? if it's only >>> that, what could cause Spark to ignore it? >>> >>> I'll appreciate any help in advance. >>> >>> -- >>> *PhD Student - EIS Group - Bonn University, Germany.* >>> *+49 1575 8482232 <%2B49%201575%208482232>* >>> >>> >> > > > > [image: What's New with Xactly] <http://www.xactlycorp.com/email-click/> > > <https://www.nyse.com/quote/XNYS:XTLY> [image: LinkedIn] > <https://www.linkedin.com/company/xactly-corporation> [image: Twitter] > <https://twitter.com/Xactly> [image: Facebook] > <https://www.facebook.com/XactlyCorp> [image: YouTube] > <http://www.youtube.com/xactlycorporation>
