you might also want to just check what file-max is, more /proc/sys/fs/file-max
I just checked on my fedora and ubuntu systems and it appears they default to 785130 and 2452636 respectively so you might not want to accidentally decrease those numbers. On 4/10/13, Andrew Purtell <[email protected]> wrote: > Correct, nproc has nothing to do with file table issues. > > I typically do something like this when setting up a node: > > echo "@hadoop soft nofile 65536" >> /etc/security/limits.conf > echo "@hadoop hard nofile 65536" >> /etc/security/limits.conf > > where all accounts launching Hadoop daemons are in the 'hadoop' group. Make > sure you've done something like this. > > You may need to increase the maximum number of file-handles that the Linux > kernel will allocate overall. Try adding this to an init script: > > sysctl -w fs.file-max=131072 > > ... for example. Or you can add "fs.file-max=131072" into /etc/sysctl.conf. > If you do that, then be sure to execute sysctl -p as root for the change to > take effect. > > > > On Tue, Apr 9, 2013 at 9:08 AM, Jean-Marc Spaggiari > <[email protected] >> wrote: > >> But there was not any trace looking like "OutOfMemoryError". nproc >> might has result with this, no? Not a SocketException? >> Anyway, I have increased it to 32768. I will see if I face that again. >> >> Thanks, >> >> JM >> >> 2013/4/9 Ted Yu <[email protected]>: >> > According to http://hbase.apache.org/book.html#ulimit , you should >> increase >> > nproc setting. >> > >> > Cheers >> > >> > On Tue, Apr 9, 2013 at 8:33 AM, Jean-Marc Spaggiari < >> [email protected] >> >> wrote: >> > >> >> Hi, >> >> >> >> I just faced an issue this morning on one of my RS. >> >> >> >> Here is an extract of the logs >> >> 2013-04-09 11:05:33,164 ERROR org.apache.hadoop.hdfs.DFSClient: >> >> Exception closing file >> >> >> >> >> /hbase/entry_proposed/ae4a5d72d4613728ddbcc5a64262371b/.tmp/ed6a0154ef714cd88faf26061cf248d3 >> >> : java.net.SocketException: Too many open files >> >> java.net.SocketException: Too many open files >> >> at sun.nio.ch.Net.socket0(Native Method) >> >> at sun.nio.ch.Net.socket(Net.java:323) >> >> at sun.nio.ch.Net.socket(Net.java:316) >> >> at >> sun.nio.ch.SocketChannelImpl.<init>(SocketChannelImpl.java:101) >> >> at >> >> >> sun.nio.ch.SelectorProviderImpl.openSocketChannel(SelectorProviderImpl.java:60) >> >> at >> >> java.nio.channels.SocketChannel.open(SocketChannel.java:142) >> >> at >> >> >> org.apache.hadoop.net.StandardSocketFactory.createSocket(StandardSocketFactory.java:58) >> >> at >> >> >> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:3423) >> >> at >> >> >> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3381) >> >> at >> >> >> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2600(DFSClient.java:2589) >> >> at >> >> >> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2829) >> >> >> >> ulimit is unlimited on all my servers. >> >> >> >> Is seems there was to many network connections opened. Is there >> >> anything HBase can handle in such scenario? It's only hadoop in the >> >> stacktrace, so I'm not sure. >> >> >> >> Can this be related to nproc? I don't think so. I have another tool >> >> running on the RS. Using low CPU, low bandwidth but making MANY >> >> network HTTP connections... >> >> >> >> Any suggestion? >> >> >> >> JM >> >> >> > > > > -- > Best regards, > > - Andy > > Problems worthy of attack prove their worth by hitting back. - Piet Hein > (via Tom White) > -- Ted.
