[
https://issues.apache.org/jira/browse/CASSANDRA-1495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12908569#action_12908569
]
Josep M. Blanquer commented on CASSANDRA-1495:
----------------------------------------------
Ah, yes, it smells quite similar. There's not much text/description in it, I
think that's why I couldn't find it when I searched around before opening the
ticket.
So, I've done some in-place patching on some of the machines, and I believe it
is exactly the same problem (they don't seem to leak anymore).
..so that 'solves' it for now I guess. This ticket should be closed or marked
as duplicate?...not sure how you guys do it in here, so I'll let you proceed
with it if you don't mind. :)
Thanks for the pointer Brandon.
> Server doesn't seem to close SSTables correctly, and ends up with too many
> file descriptors open
> ------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-1495
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1495
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7 beta 1
> Environment: 0.7 beta1. Built from:
> http://www.apache.org/dyn/closer.cgi?path=/cassandra/0.7.0/apache-cassandra-0.7.0-beta1-src.tar.gz
> using dpkg-buildpackage from it.
> Reporter: Josep M. Blanquer
> Fix For: 0.7 beta 2
>
>
> Cassandra server accumulates many open file descriptors as the time
> progresses. Obviously when it hits whatever limit the system allows, the
> service stops accepting messages.
> The exact trace is:
> WARN 05:41:47,929 Transport error occurred during acceptance of message.
> org.apache.thrift.transport.TTransportException: java.net.SocketException:
> Too many open files
> at
> org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:124)
> at
> org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:35)
> at
> org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31)
> at
> org.apache.cassandra.thrift.CustomTThreadPoolServer.serve(CustomTThreadPoolServer.java:98)
> at
> org.apache.cassandra.thrift.CassandraDaemon.start(CassandraDaemon.java:210)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.commons.daemon.support.DaemonLoader.start(DaemonLoader.java:177)
> Caused by: java.net.SocketException: Too many open files
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
> at java.net.ServerSocket.accept(ServerSocket.java:421)
> at
> org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:119)
> ... 9 more
> I've increased the ulimit to 8K from the standard 1K ... but still gets hit.
> So it seems that basically there are many fd's that never get closed.
> lsof shows that there are many fd's hanging on to SSTables...albeit it's only
> a small subset of unique files. In my case there were only about 4 distinct
> SSTable files...but kept opened hundreds of time each.
> All of these files seem to be "Data" files....no Filter or Index ones (as far
> as I remember)
> In case it matters: DiskAccessMode 'auto' determined to be standard,
> indexAccessMode is standard
> The service doesn't have a high rate of writes at the moment...I have 3
> nodes, 1 being receiving mostly the thrift entry point. I see the problem
> being more acute on the 2 nodes that instead of being contacted by the
> clients, they just get the writes from the coordinator...(I have RF=2).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
