The symptoms are consistent with https://issues.apache.org/jira/browse/CASSANDRA-533. I agree with Jonathan that the fix for that is in Trunk and not in 0.4.2. There's a code fix given in the bug, if you want to hack source and recompile.
Tim Freeman Email: [email protected] Desk in Palo Alto: (650) 857-2581 Home: (408) 774-1298 Cell: (408) 348-7536 (No reception business hours Monday, Tuesday, and Thursday; call my desk instead.) -----Original Message----- From: Jonathan Ellis [mailto:[email protected]] Sent: Thursday, November 12, 2009 6:23 PM To: [email protected] Subject: Re: Leaked FDs This is fixed in trunk and the 0.4 branch but did not make 0.4.2. On Thu, Nov 12, 2009 at 6:28 PM, elsif <[email protected]> wrote: > Hello. I am testing out Cassandra and ran out of file descriptors while > attempting to scan through the key space. I have ~180 million unique > keys, 10 nodes, and a replication factor of 3. > > Cassandra reported the following: > > WARN [GMFD:1] 2009-11-12 16:07:24,961 MessagingService.java (line 393) > Exception was generated at : 11/12/2009 16:07:24 on thread GMFD:1 > Too many open files > java.net.SocketException: Too many open files > at sun.nio.ch.Net.socket0(Native Method) > at sun.nio.ch.Net.socket(Unknown Source) > at sun.nio.ch.DatagramChannelImpl.<init>(Unknown Source) > at sun.nio.ch.SelectorProviderImpl.openDatagramChannel(Unknown > Source) > at java.nio.channels.DatagramChannel.open(Unknown Source) > at > org.apache.cassandra.net.UdpConnection.init(UdpConnection.java:49) > at > org.apache.cassandra.net.MessagingService.sendUdpOneWay(MessagingService.java:388) > at > org.apache.cassandra.gms.GossipDigestSynVerbHandler.doVerb(Gossiper.java:889) > at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:38) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown > Source) > at java.lang.Thread.run(Unknown Source) > > Lsof reports that the java process has 65486 files open (I have ulimit > -n 65535 set in cassandra.in.sh). Many of the lsof entries include a > trailing '(deleted)' comment after the file path. > > This appears to be similar to CASSANDRA-283. Anyone have a work around > for this? Would a forced GC take care of the ones marked deleted? > > Here is my sample code to count the number of keys: > > public class CClient > { > public static void main(String[] args) > throws TException, InvalidRequestException, UnavailableException, > UnsupportedEncodingException, NotFoundException > { > TTransport tr = new TSocket("localhost", 9160); > TProtocol proto = new TBinaryProtocol(tr); > Cassandra.Client client = new Cassandra.Client(proto); > tr.open(); > > int count = 0; > int block = 1000; > String key = " "; > > while (true) > { > List<String> list = client.get_key_range("Keyspace1", > "Standard1", key, "~", block, ConsistencyLevel.ONE); > int size = list.size(); > if (size == 0) > break; > count += size; > key = list.get(size - 1) + '~'; > System.out.println("Count: " + Integer.toString(count)); > } > tr.close(); > } > } > >
