JVM OOM on node startup
Hello again. We have 3 nodes and were testing what happens when a node goes down. There is roughly 10gb of data on each node. The node we simulated dieing was working just fine under the load. Then we killed it. The ring performed admirably, But upon restarting the node it dies every time of JVM OOM errors. I have forced a JVM heap size of 1024mb in the startup file. (did this because adaptive heap size was causing oom errors with normal usage.) The machines are 2 core 4gb ram vm's. I've read the Riptano troubleshooting guide... http://www.riptano.com/docs/0.6/troubleshooting/index#nodes-are-dying-with-oom-errors But im not sure if these apply in this case since it is only dieing on startup. Here is a link to the startup logs as it dies. http://pastebin.com/BEXeVvCX Thank you for any help you can provide.
Re: JVM OOM on node startup
Looks like it's trying to load your row cache and running out of memory, probably because you reduced the memory. The cassandra-env.sh script would have been giving it 2GB.1Gb heap is probably going to be to small.Was this the same error you were getting before you reduced the memory ?Try deleting the caches, the path is specified by the saved_caches_directory setting in cassandra.yaml.Also what version are you using ? The errorCaused by: javax.management.AttributeNotFoundException: No such attribute: ActiveCount reminds me of a problem in beta 1.Hope that helps.AaronOn 01 Dec, 2010,at 09:28 AM, Brayton Thompson thomp...@grnoc.iu.edu wrote:Hello again. We have 3 nodes and were testing what happens when a node goes down. There is roughly 10gb of data on each node. The node we "simulated" dieing was working just fine under the load. Then we killed it. The ring performed admirably, But upon restarting the node it dies every time of JVM OOM errors. I have forced a JVM heap size of 1024mb in the startup file. (did this because adaptive heap size was causing oom errors with normal usage.) The machines are 2 core 4gb ram vm's. I've read the Riptano troubleshooting guide... http://www.riptano.com/docs/0.6/troubleshooting/index#nodes-are-dying-with-oom-errors But im not sure if these apply in this case since it is only dieing on startup. Here is a link to the startup logs as it dies. http://pastebin.com/BEXeVvCX Thank you for any help you can provide.
Re: JVM OOM on node startup
If you're getting OOM with adaptive heap size of 1GB, reducing it to 1GB is not going to make things better. :) On Tue, Nov 30, 2010 at 2:28 PM, Brayton Thompson thomp...@grnoc.iu.edu wrote: Hello again. We have 3 nodes and were testing what happens when a node goes down. There is roughly 10gb of data on each node. The node we simulated dieing was working just fine under the load. Then we killed it. The ring performed admirably, But upon restarting the node it dies every time of JVM OOM errors. I have forced a JVM heap size of 1024mb in the startup file. (did this because adaptive heap size was causing oom errors with normal usage.) The machines are 2 core 4gb ram vm's. I've read the Riptano troubleshooting guide... http://www.riptano.com/docs/0.6/troubleshooting/index#nodes-are-dying-with-oom-errors But im not sure if these apply in this case since it is only dieing on startup. Here is a link to the startup logs as it dies. http://pastebin.com/BEXeVvCX Thank you for any help you can provide. -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of Riptano, the source for professional Cassandra support http://riptano.com
