For cassandra driver,session is just like database connection pool,it maybe contains many tcp connections,if you create a new session every time,more and more tcp connections will be created,till surpass the max file description limit of os.
You should create one session,use it repeatedly ,session can manage connections automatically,create new connection or close old connection for your requests. 在 2014年8月9日，6:52，Redmumba <redmu...@gmail.com> 写道： > Just to chime in, I also ran into this issue when I was migrating to the > Datastax client. Instead of reusing the session, I was opening a new session > each time. For some reason, even though I was still closing the session on > the client side, I was getting the same error. > > Plus, the only way I could recover was by restarting Cassandra. I did not > really see the connections timeout over a period of a few minutes. > > Andrew > > On Aug 8, 2014 3:19 PM, "Andrey Ilinykh" <ailin...@gmail.com> wrote: > You may have this problem if your client doesn't reuse the connection but > opens new every type. So, run netstat and check the number of established > connections. This number should not be big. > > Thank you, > Andrey > > > On Fri, Aug 8, 2014 at 12:35 PM, Marcelo Elias Del Valle > <marc...@s1mbi0se.com.br> wrote: > Hi, > > I am using Cassandra 2.0.9 running on Debian Wheezy, and I am having "too > many open files" exceptions when I try to perform a large number of > operations in my 10 node cluster. > > I saw the documentation > http://www.datastax.com/documentation/cassandra/2.0/cassandra/troubleshooting/trblshootTooManyFiles_r.html > and I have set everything to the recommended settings, but I keep getting > the errors. > > In the documentation it says: "Another, much less likely possibility, is a > file descriptor leak in Cassandra. Run lsof -n | grep java to check that the > number of file descriptors opened by Java is reasonable and reports the error > if the number is greater than a few thousand." > > I guess it's not the case, or else a lot of people would be complaining about > it, but I am not sure what I could do to solve the problem. > > Any hint about how to solve it? > > My client is written in python and uses Cassandra Python Driver. Here are the > exceptions I am having in the client: > [s1log] 2014-08-08 12:16:09,631 - cassandra.pool - WARNING - Error attempting > to reconnect to 200.200.200.151, scheduling retry in 600.0 seconds: [Errno > 24] Too many open files > [s1log] 2014-08-08 12:16:09,632 - cassandra.pool - WARNING - Error attempting > to reconnect to 200.200.200.142, scheduling retry in 600.0 seconds: [Errno > 24] Too many open files > [s1log] 2014-08-08 12:16:09,633 - cassandra.pool - WARNING - Error attempting > to reconnect to 200.200.200.143, scheduling retry in 600.0 seconds: [Errno > 24] Too many open files > [s1log] 2014-08-08 12:16:09,634 - cassandra.pool - WARNING - Error attempting > to reconnect to 200.200.200.142, scheduling retry in 600.0 seconds: [Errno > 24] Too many open files > [s1log] 2014-08-08 12:16:09,634 - cassandra.pool - WARNING - Error attempting > to reconnect to 200.200.200.145, scheduling retry in 600.0 seconds: [Errno > 24] Too many open files > [s1log] 2014-08-08 12:16:09,635 - cassandra.pool - WARNING - Error attempting > to reconnect to 200.200.200.144, scheduling retry in 600.0 seconds: [Errno > 24] Too many open files > [s1log] 2014-08-08 12:16:09,635 - cassandra.pool - WARNING - Error attempting > to reconnect to 200.200.200.148, scheduling retry in 600.0 seconds: [Errno > 24] Too many open files > [s1log] 2014-08-08 12:16:09,732 - cassandra.pool - WARNING - Error attempting > to reconnect to 200.200.200.146, scheduling retry in 600.0 seconds: [Errno > 24] Too many open files > [s1log] 2014-08-08 12:16:09,733 - cassandra.pool - WARNING - Error attempting > to reconnect to 200.200.200.77, scheduling retry in 600.0 seconds: [Errno 24] > Too many open files > [s1log] 2014-08-08 12:16:09,734 - cassandra.pool - WARNING - Error attempting > to reconnect to 200.200.200.76, scheduling retry in 600.0 seconds: [Errno 24] > Too many open files > [s1log] 2014-08-08 12:16:09,734 - cassandra.pool - WARNING - Error attempting > to reconnect to 200.200.200.75, scheduling retry in 600.0 seconds: [Errno 24] > Too many open files > [s1log] 2014-08-08 12:16:09,735 - cassandra.pool - WARNING - Error attempting > to reconnect to 200.200.200.142, scheduling retry in 600.0 seconds: [Errno > 24] Too many open files > [s1log] 2014-08-08 12:16:09,736 - cassandra.pool - WARNING - Error attempting > to reconnect to 200.200.200.185, scheduling retry in 600.0 seconds: [Errno > 24] Too many open files > [s1log] 2014-08-08 12:16:09,942 - cassandra.pool - WARNING - Error attempting > to reconnect to 200.200.200.144, scheduling retry in 512.0 seconds: Timed out > connecting to 200.200.200.144 > [s1log] 2014-08-08 12:16:09,998 - cassandra.pool - WARNING - Error attempting > to reconnect to 200.200.200.77, scheduling retry in 512.0 seconds: Timed out > connecting to 200.200.200.77 > > > And here is the exception I am having in the server: > > WARN [Native-Transport-Requests:163] 2014-08-08 14:27:30,499 > BatchStatement.java (line 223) Batch of prepared statements for > [identification.entity_lookup, identification.entity] is of size 25216, > exceeding specified threshold of 5120 by 20096. > ERROR [Native-Transport-Requests:150] 2014-08-08 14:27:31,611 > ErrorMessage.java (line 222) Unexpected exception during request > java.io.IOException: Connection reset by peer > at sun.nio.ch.FileDispatcherImpl.read0(Native Method) > at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) > at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) > at sun.nio.ch.IOUtil.read(IOUtil.java:192) > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:375) > at > org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:64) > at > org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109) > at > org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312) > at > org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90) > at > org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > > Here is the amount of open files before and after I restart Cassandra: > > root@h:/etc/security/limits.d# lsof -n | grep java | wc -l > 936580 > root@h:/etc/security/limits.d# sudo service cassandra restart > [ ok ] Restarting Cassandra: cassandra. > root@h:/etc/security/limits.d# lsof -n | grep java | wc -l > 80295 > > > Best regards, > Marcelo Valle. >
