If you don’t have row caching turned on, you can try changing settings to get your memtables to flush faster (read more at the datastax documention at [1]). Btw, if you are using a pre-1.2 version, your bloom filter might come into play as well.
All said and done, if your write request rate is too high for your ring to handle, you just can’t flush fast enough, and you’ll need to move to physical machines and/or add nodes to your ring. -Arindam [1] http://www.datastax.com/documentation/cassandra/2.0/cassandra/troubleshooting/trblshootDyingNodes_r.html From: Gary Zhao [mailto:garyz...@gmail.com] Sent: Monday, April 28, 2014 5:34 PM To: user@cassandra.apache.org Subject: Re: java.lang.OutOfMemoryError: Java heap space Yes, they are virtual machines, but we are using KVM. Is there any solutions for this issue or we should use physical machines? On Mon, Apr 28, 2014 at 10:38 AM, Prem Yadav <ipremya...@gmail.com<mailto:ipremya...@gmail.com>> wrote: Are the virtual machines? The last time I had this issues was because of VMWare "ballooning". If not, what versions of Cassandra and Java are you using? On Mon, Apr 28, 2014 at 6:30 PM, Gary Zhao <garyz...@gmail.com<mailto:garyz...@gmail.com>> wrote: BTW, the CPU usage on this node is pretty high, but data size is pretty small. PID USERNAME THR PRI NICE SIZE RES SHR STATE TIME CPU COMMAND 28674 cassandr 89 25 0 9451M 8970M 525M sleep 32.1H 329% java UN 8.92 GB 256 35.4% c2d9d02e-bdb3-47cb-af1b-eabc2eeb503b rack1 UN 5.32 GB 256 33.1% 874a269f-96bc-4db6-9d15-7cac6771a2ea rack1 DN 339.94 MB 256 31.5% fba5b071-aee1-451f-ae42-49c3bbe70bbe rack1 On Mon, Apr 28, 2014 at 10:25 AM, Gary Zhao <garyz...@gmail.com<mailto:garyz...@gmail.com>> wrote: Hello I have a three nodes cluster. I noticed one node was always down. Restarting Cassandra fixes it but it will go down again after a couple of days. I'm pretty new to Cassandra so I'm wondering how I should troubleshoot it. Logs is as below. INFO [StorageServiceShutdownHook] 2014-04-28 13:21:05,091 ThriftServer.java (line 141) Stop listening to thrift clients ERROR [GossipStage:4] 2014-04-28 13:11:59,877 CassandraDaemon.java (line 196) Exception in thread Thread[GossipStage:4,5,main] java.lang.OutOfMemoryError: Java heap space ERROR [ACCEPT-/10.20.132.44<http://10.20.132.44>] 2014-04-28 13:06:10,261 CassandraDaemon.java (line 196) Exception in thread Thread[ACCEPT-/10.20.132.44<http://10.20.132.44>,5,main] java.lang.OutOfMemoryError: Java heap space ERROR [GossipStage:3] 2014-04-28 12:54:02,116 CassandraDaemon.java (line 196) Exception in thread Thread[GossipStage:3,5,main] java.lang.OutOfMemoryError: Java heap space ERROR [GossipStage:2] 2014-04-28 12:52:27,644 CassandraDaemon.java (line 196) Exception in thread Thread[GossipStage:2,5,main] java.lang.OutOfMemoryError: Java heap space ERROR [Thread-222] 2014-04-28 12:50:18,689 CassandraDaemon.java (line 196) Exception in thread Thread[Thread-222,5,main] java.lang.OutOfMemoryError: Java heap space ERROR [GossipTasks:1] 2014-04-28 12:47:12,879 CassandraDaemon.java (line 196) Exception in thread Thread[GossipTasks:1,5,main] java.lang.OutOfMemoryError: Java heap space ERROR [GossipTasks:1] 2014-04-28 13:24:59,113 CassandraDaemon.java (line 196) Exception in thread Thread[GossipTasks:1,5,main] java.lang.IllegalThreadStateException at java.lang.Thread.start(Thread.java:704) at org.apache.cassandra.service.CassandraDaemon$2.uncaughtException(CassandraDaemon.java:202) at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.handleOrLog(DebuggableThreadPoolExecutor.java:220) at org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:79) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) INFO [StorageServiceShutdownHook] 2014-04-28 13:25:35,105 Server.java (line 181) Stop listening for CQL clients INFO [StorageServiceShutdownHook] 2014-04-28 13:25:35,105 Gossiper.java (line 1251) Announcing shutdown INFO [StorageServiceShutdownHook] 2014-04-28 13:26:12,524 MessagingService.java (line 667) Waiting for messaging service to quiesce Thanks Gary
