Hi Zhiyan, Silly question but are you sure your heap settings are actually being applied? "697,236,904 (51.91%)" would represent a sub-2GB heap. What's the real memory usage for Java when this crash happens?
Other thing to look into might be memtable_heap_space_in_mb, as it looks like you're using onheap memtables. This will be 1/4 of your heap by default. Assuming your heap settings are actually being applied, if you run through this space you may not have enough flushing resources. memtable_flush_Writers defaults to a somewhat low number which may not be enough for this use case. On Fri, May 20, 2016 at 10:02 PM, Zhiyan Shao <zhiyan.s...@gmail.com> wrote: > Hi, we see the following OOM crash while doing heavy write loading > testing. Has anybody seen this kind of crash? We are using G1GC with 32GB > heap size out of 128GB system memory. Eclipse Memory Analyzer shows the > following: > > One instance of *"org.apache.cassandra.db.ColumnFamilyStore"* loaded by > *"sun.misc.Launcher$AppClassLoader > @ 0x8d800898"* occupies *697,236,904 (51.91%)* bytes. The memory is > accumulated in one instance of > *"java.util.concurrent.ConcurrentSkipListMap$HeadIndex"* loaded by *"<system > class loader>"*. > > *Keywords* > > java.util.concurrent.ConcurrentSkipListMap$HeadIndex > > sun.misc.Launcher$AppClassLoader @ 0x8d800898 > > org.apache.cassandra.db.ColumnFamilyStore > > Cassandra log: > > > ERROR 00:23:24 JVM state determined to be unstable. Exiting forcefully > due to: > java.lang.OutOfMemoryError: Java heap space > at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57) ~[na:1.8.0_74] > at java.nio.ByteBuffer.allocate(ByteBuffer.java:335) ~[na:1.8.0_74] > at > org.apache.cassandra.utils.memory.SlabAllocator.getRegion(SlabAllocator.java: > 137) ~[apache-cassandra-2.2.5.jar:2.2.5] > at > org.apache.cassandra.utils.memory.SlabAllocator.allocate(SlabAllocator.java: > 97) ~[apache-cassandra-2.2.5.jar:2.2.5] > at > org.apache.cassandra.utils.memory.ContextAllocator.allocate(ContextAllocator.java: > 57) ~[apache-cassandra-2.2.5.jar:2.2.5] > at org.apache.cassandra.utils.memory.ContextAllocator.clone > (ContextAllocator.java:47) ~[apache-cassandra-2.2.5.jar:2.2.5] > at org.apache.cassandra.utils.memory.MemtableBufferAllocator.clone > (MemtableBufferAllocator.java:61) ~[apache-cassandra-2.2.5.jar:2.2.5] > at org.apache.cassandra.db.Memtable.put(Memtable.java:212) > ~[apache-cassandra-2.2.5.jar:2.2.5] > at org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java: > 1249) ~[apache-cassandra-2.2.5.jar:2.2.5] > at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:406) > ~[apache-cassandra-2.2.5.jar:2.2.5] > at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:366) > ~[apache-cassandra-2.2.5.jar:2.2.5] > at org.apache.cassandra.db.Mutation.apply(Mutation.java:214) > ~[apache-cassandra-2.2.5.jar:2.2.5] > at > org.apache.cassandra.db.MutationVerbHandler.doVerb(MutationVerbHandler.java: > 50) ~[apache-cassandra-2.2.5.jar:2.2.5] > at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java: > 67) ~[apache-cassandra-2.2.5.jar:2.2.5] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > ~[na:1.8.0_74] > at > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java: > 164) ~[apache-cassandra-2.2.5.jar:2.2.5] > at > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java: > 136) [apache-cassandra-2.2.5.jar:2.2.5] > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) > [apache-cassandra-2.2.5.jar:2.2.5] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74] > > Thanks, > Zhiyan >
