Maybe the drivers should have two modes: few sessions, and lots of sessions.
The former would give you a developer-friendly driver error if you leave more
than say a dozen or two dozen sessions open (or whatever is considered a best
practice for parallel threads in a client), on the theory that you probably
used the anti-pattern of failing to reuse sessions. The latter would be more
for expert apps that have some good reason for having hundreds or thousands of
simultaneous sessions open. Whether the latter also has some (configurable)
limit that is simply a lot higher than the former or is unlimited, is probably
not so important. Or, maybe, simply have a single limit, without the modes and
default it to 10 or 25 or some other relatively low number for “normal” apps.
This would be more developer-friendly, for both new and “normal” developers...
I think.
-- Jack Krupansky
From: Marcelo Elias Del Valle
Sent: Saturday, August 9, 2014 12:41 AM
To: user@cassandra.apache.org
Subject: Re: too many open files
Indeed, that was my mistake, that was exactly what we were doing in the code.
[]s
2014-08-09 0:56 GMT-03:00 Brian Zhang <yikebo...@gmail.com>:
For cassandra driver,session is just like database connection pool,it maybe
contains many tcp connections,if you create a new session every time,more and
more tcp connections will be created,till surpass the max file description
limit of os.
You should create one session,use it repeatedly ,session can manage
connections automatically,create new connection or close old connection for
your requests.
在 2014年8月9日，6:52，Redmumba <redmu...@gmail.com> 写道：
Just to chime in, I also ran into this issue when I was migrating to the
Datastax client. Instead of reusing the session, I was opening a new session
each time. For some reason, even though I was still closing the session on the
client side, I was getting the same error.
Plus, the only way I could recover was by restarting Cassandra. I did not
really see the connections timeout over a period of a few minutes.
Andrew
On Aug 8, 2014 3:19 PM, "Andrey Ilinykh" <ailin...@gmail.com> wrote:
You may have this problem if your client doesn't reuse the connection but
opens new every type. So, run netstat and check the number of established
connections. This number should not be big.
Thank you,
Andrey
On Fri, Aug 8, 2014 at 12:35 PM, Marcelo Elias Del Valle
<marc...@s1mbi0se.com.br> wrote:
Hi,
I am using Cassandra 2.0.9 running on Debian Wheezy, and I am having
"too many open files" exceptions when I try to perform a large number of
operations in my 10 node cluster.
I saw the documentation
http://www.datastax.com/documentation/cassandra/2.0/cassandra/troubleshooting/trblshootTooManyFiles_r.html
and I have set everything to the recommended settings, but I keep getting the
errors.
In the documentation it says: "Another, much less likely possibility,
is a file descriptor leak in Cassandra. Run lsof -n | grep java to check that
the number of file descriptors opened by Java is reasonable and reports the
error if the number is greater than a few thousand."
I guess it's not the case, or else a lot of people would be complaining
about it, but I am not sure what I could do to solve the problem.
Any hint about how to solve it?
My client is written in python and uses Cassandra Python Driver. Here
are the exceptions I am having in the client:
[s1log] 2014-08-08 12:16:09,631 - cassandra.pool - WARNING - Error
attempting to reconnect to 200.200.200.151, scheduling retry in 600.0 seconds:
[Errno 24] Too many open files
[s1log] 2014-08-08 12:16:09,632 - cassandra.pool - WARNING - Error
attempting to reconnect to 200.200.200.142, scheduling retry in 600.0 seconds:
[Errno 24] Too many open files
[s1log] 2014-08-08 12:16:09,633 - cassandra.pool - WARNING - Error
attempting to reconnect to 200.200.200.143, scheduling retry in 600.0 seconds:
[Errno 24] Too many open files
[s1log] 2014-08-08 12:16:09,634 - cassandra.pool - WARNING - Error
attempting to reconnect to 200.200.200.142, scheduling retry in 600.0 seconds:
[Errno 24] Too many open files
[s1log] 2014-08-08 12:16:09,634 - cassandra.pool - WARNING - Error
attempting to reconnect to 200.200.200.145, scheduling retry in 600.0 seconds:
[Errno 24] Too many open files
[s1log] 2014-08-08 12:16:09,635 - cassandra.pool - WARNING - Error
attempting to reconnect to 200.200.200.144, scheduling retry in 600.0 seconds:
[Errno 24] Too many open files
[s1log] 2014-08-08 12:16:09,635 - cassandra.pool - WARNING - Error
attempting to reconnect to 200.200.200.148, scheduling retry in 600.0 seconds:
[Errno 24] Too many open files
[s1log] 2014-08-08 12:16:09,732 - cassandra.pool - WARNING - Error
attempting to reconnect to 200.200.200.146, scheduling retry in 600.0 seconds:
[Errno 24] Too many open files
[s1log] 2014-08-08 12:16:09,733 - cassandra.pool - WARNING - Error
attempting to reconnect to 200.200.200.77, scheduling retry in 600.0 seconds:
[Errno 24] Too many open files
[s1log] 2014-08-08 12:16:09,734 - cassandra.pool - WARNING - Error
attempting to reconnect to 200.200.200.76, scheduling retry in 600.0 seconds:
[Errno 24] Too many open files
[s1log] 2014-08-08 12:16:09,734 - cassandra.pool - WARNING - Error
attempting to reconnect to 200.200.200.75, scheduling retry in 600.0 seconds:
[Errno 24] Too many open files
[s1log] 2014-08-08 12:16:09,735 - cassandra.pool - WARNING - Error
attempting to reconnect to 200.200.200.142, scheduling retry in 600.0 seconds:
[Errno 24] Too many open files
[s1log] 2014-08-08 12:16:09,736 - cassandra.pool - WARNING - Error
attempting to reconnect to 200.200.200.185, scheduling retry in 600.0 seconds:
[Errno 24] Too many open files
[s1log] 2014-08-08 12:16:09,942 - cassandra.pool - WARNING - Error
attempting to reconnect to 200.200.200.144, scheduling retry in 512.0 seconds:
Timed out connecting to 200.200.200.144
[s1log] 2014-08-08 12:16:09,998 - cassandra.pool - WARNING - Error
attempting to reconnect to 200.200.200.77, scheduling retry in 512.0 seconds:
Timed out connecting to 200.200.200.77
And here is the exception I am having in the server:
WARN [Native-Transport-Requests:163] 2014-08-08 14:27:30,499
BatchStatement.java (line 223) Batch of prepared statements for
[identification.entity_lookup, identification.entity] is of size 25216,
exceeding specified threshold of 5120 by 20096.
ERROR [Native-Transport-Requests:150] 2014-08-08 14:27:31,611
ErrorMessage.java (line 222) Unexpected exception during request
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:375)
at
org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:64)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
at
org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Here is the amount of open files before and after I restart Cassandra:
root@h:/etc/security/limits.d# lsof -n | grep java | wc -l
936580
root@h:/etc/security/limits.d# sudo service cassandra restart
[ ok ] Restarting Cassandra: cassandra.
root@h:/etc/security/limits.d# lsof -n | grep java | wc -l
80295
Best regards,
Marcelo Valle.
