Are you using apache or Datastax cassandra? The datastax distribution ups the file handle limit to 100000. That number's hard to exceed.
On Fri, Aug 8, 2014 at 1:35 PM, Marcelo Elias Del Valle < marc...@s1mbi0se.com.br> wrote: > Hi, > > I am using Cassandra 2.0.9 running on Debian Wheezy, and I am having "too > many open files" exceptions when I try to perform a large number of > operations in my 10 node cluster. > > I saw the documentation > http://www.datastax.com/documentation/cassandra/2.0/cassandra/troubleshooting/trblshootTooManyFiles_r.html > and I have set everything to the recommended settings, but I keep getting > the errors. > > In the documentation it says: "Another, much less likely possibility, is > a file descriptor leak in Cassandra. Run lsof -n | grep java to check > that the number of file descriptors opened by Java is reasonable and > reports the error if the number is greater than a few thousand." > > I guess it's not the case, or else a lot of people would be complaining > about it, but I am not sure what I could do to solve the problem. > > Any hint about how to solve it? > > My client is written in python and uses Cassandra Python Driver. Here are > the exceptions I am having in the client: > [s1log] 2014-08-08 12:16:09,631 - cassandra.pool - WARNING - Error > attempting to reconnect to 200.200.200.151, scheduling retry in 600.0 > seconds: [Errno 24] Too many open files > [s1log] 2014-08-08 12:16:09,632 - cassandra.pool - WARNING - Error > attempting to reconnect to 200.200.200.142, scheduling retry in 600.0 > seconds: [Errno 24] Too many open files > [s1log] 2014-08-08 12:16:09,633 - cassandra.pool - WARNING - Error > attempting to reconnect to 200.200.200.143, scheduling retry in 600.0 > seconds: [Errno 24] Too many open files > [s1log] 2014-08-08 12:16:09,634 - cassandra.pool - WARNING - Error > attempting to reconnect to 200.200.200.142, scheduling retry in 600.0 > seconds: [Errno 24] Too many open files > [s1log] 2014-08-08 12:16:09,634 - cassandra.pool - WARNING - Error > attempting to reconnect to 200.200.200.145, scheduling retry in 600.0 > seconds: [Errno 24] Too many open files > [s1log] 2014-08-08 12:16:09,635 - cassandra.pool - WARNING - Error > attempting to reconnect to 200.200.200.144, scheduling retry in 600.0 > seconds: [Errno 24] Too many open files > [s1log] 2014-08-08 12:16:09,635 - cassandra.pool - WARNING - Error > attempting to reconnect to 200.200.200.148, scheduling retry in 600.0 > seconds: [Errno 24] Too many open files > [s1log] 2014-08-08 12:16:09,732 - cassandra.pool - WARNING - Error > attempting to reconnect to 200.200.200.146, scheduling retry in 600.0 > seconds: [Errno 24] Too many open files > [s1log] 2014-08-08 12:16:09,733 - cassandra.pool - WARNING - Error > attempting to reconnect to 200.200.200.77, scheduling retry in 600.0 > seconds: [Errno 24] Too many open files > [s1log] 2014-08-08 12:16:09,734 - cassandra.pool - WARNING - Error > attempting to reconnect to 200.200.200.76, scheduling retry in 600.0 > seconds: [Errno 24] Too many open files > [s1log] 2014-08-08 12:16:09,734 - cassandra.pool - WARNING - Error > attempting to reconnect to 200.200.200.75, scheduling retry in 600.0 > seconds: [Errno 24] Too many open files > [s1log] 2014-08-08 12:16:09,735 - cassandra.pool - WARNING - Error > attempting to reconnect to 200.200.200.142, scheduling retry in 600.0 > seconds: [Errno 24] Too many open files > [s1log] 2014-08-08 12:16:09,736 - cassandra.pool - WARNING - Error > attempting to reconnect to 200.200.200.185, scheduling retry in 600.0 > seconds: [Errno 24] Too many open files > [s1log] 2014-08-08 12:16:09,942 - cassandra.pool - WARNING - Error > attempting to reconnect to 200.200.200.144, scheduling retry in 512.0 > seconds: Timed out connecting to 200.200.200.144 > [s1log] 2014-08-08 12:16:09,998 - cassandra.pool - WARNING - Error > attempting to reconnect to 200.200.200.77, scheduling retry in 512.0 > seconds: Timed out connecting to 200.200.200.77 > > > And here is the exception I am having in the server: > > WARN [Native-Transport-Requests:163] 2014-08-08 14:27:30,499 > BatchStatement.java (line 223) Batch of prepared statements for > [identification.entity_lookup, identification.entity] is of size 25216, > exceeding specified threshold of 5120 by 20096. > ERROR [Native-Transport-Requests:150] 2014-08-08 14:27:31,611 > ErrorMessage.java (line 222) Unexpected exception during request > java.io.IOException: Connection reset by peer > at sun.nio.ch.FileDispatcherImpl.read0(Native Method) > at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) > at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) > at sun.nio.ch.IOUtil.read(IOUtil.java:192) > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:375) > at > org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:64) > at > org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109) > at > org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312) > at > org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90) > at > org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > > Here is the amount of open files before and after I restart Cassandra: > > root@h:/etc/security/limits.d# lsof -n | grep java | wc -l > 936580 > root@h:/etc/security/limits.d# sudo service cassandra restart > [ ok ] Restarting Cassandra: cassandra. > root@h:/etc/security/limits.d# lsof -n | grep java | wc -l > 80295 > > > Best regards, > Marcelo Valle. >
