> Cassandra version is : apache-cassandra-1.2.4 The latest 1.2 version is 1.2.13, you really should be on that.
> commitlog_total_space_in_mb: 16 > commitlog_segment_size_in_mb: 16 Reducing the total commit log size to 16 MB is a very bad idea, you should return it to 4096 and the segment size to 32. The commit log is kept on disk and has no impact on the memory footprint. Reducing the size will cause much more disk IO. It’s kind of unusual to go OOM in 1.2+, but I’ve seen it happen with large number of SSTables (30k+) and LCS. Also wide rows, or lots of tombstones, and bad queries can result in a lot of premature tenuring. Finally custom comparators can create a lot of garbage or a low powered CPU may not be able to keep up. How many cores do you have ? You may want to make these changes to reduce how quickly objects are tenured, also pay attention to how low the total heap use get’s to after CMS. JVM_OPTS="$JVM_OPTS -XX:SurvivorRatio=4" JVM_OPTS="$JVM_OPTS -XX:MaxTenuringThreshold=2" JVM_OPTS="$JVM_OPTS -XX:CMSInitiatingOccupancyFraction=50” Hope that helps. ----------------- Aaron Morton New Zealand @aaronmorton Co-Founder & Principal Consultant Apache Cassandra Consulting http://www.thelastpickle.com On 19/12/2013, at 4:47 pm, Lee Mighdoll <l...@underneath.ca> wrote: > I'd suggest setting some cassandra jvm parameters so that you can analyze a > heap dump and peek through the gc logs. That'll give you some clues e.g. if > the memory problem is growing steadily or suddenly, and clues from a peek at > which object are using the memory. > > -XX:+HeapDumpOnOutOfMemoryError > > And if you don't want to wait six days for another failure, you can collect a > heap sooner with jmap -F. > > -Xloggc:/path/to/where/to/put/the/gc.log > -XX:+PrintGCDetails > -XX:+PrintGCDateStamps > -XX:+PrintHeapAtGC > -XX:+PrintTenuringDistribution > -XX:+PrintGCApplicationStoppedTime > -XX:+PrintPromotionFailure > > Cheers, > Lee > > > > On Wed, Dec 18, 2013 at 6:52 PM, Shammi Jayasinghe <sha...@wso2.com> wrote: > Hi, > > > We are facing with a problem on Cassandra tuning. In that we have faced with > following OOM scenario[1], after running the system for 6 days. We have tuned > the cassandra with following values. These values also obtained by going > through huge number of testing cycles. But still it has gone OOM. I would > like to know if someone can help on identifying tuning parameters. > > In this server , we have given 6GB for the Xmx value and the total memory in > the server is 8GB. Cassandra version is : apache-cassandra-1.2.4 > > Tuning parameters: > flush_largest_memtables_at: 0.5 > reduce_cache_sizes_at: 0.85 > reduce_cache_capacity_to: 0.6 > commitlog_total_space_in_mb: 16 > commitlog_segment_size_in_mb: 16 > > > > As i mentioned in the above parameters ( Flush_largest_memtable_at to 0,5) , > i feel that it has not be affected to the server. Is there any way that we > can check whether it is affected as expected to the server ? > > > > [1]WARN 19:16:50,355 Heap is 0.9971737408184552 full. You may need to reduce > memtable and/or cache sizes. Cassandra will now flush up to the two largest > memtables to free up memory. Adjust flush_largest_memtables_at threshold in > cassandra.yaml if you don't want Cassandra to do this automatically > WARN 19:18:19,784 Flushing CFS(Keyspace='QpidKeySpace', > ColumnFamily='DestinationSubscriptionsCountRow') to relieve memory pressure > ERROR 19:20:50,316 Exception in thread Thread[ReadStage:63,5,main] > java.lang.OutOfMemoryError: Java heap space > at java.nio.ByteBuffer.wrap(ByteBuffer.java:350) > at java.nio.ByteBuffer.wrap(ByteBuffer.java:373) > at > org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:391) > at > org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:392) > at > org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:371) > at > org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:84) > at > org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:73) > at > org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.getNextBlock(IndexedSliceReader.java:370) > at > org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.fetchMoreData(IndexedSliceReader.java:325) > at > org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:151) > at > org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:48) > at > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) > at > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) > at > org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:90) > at > org.apache.cassandra.db.filter.QueryFilter$2.getNext(QueryFilter.java:171) > at > org.apache.cassandra.db.filter.QueryFilter$2.hasNext(QueryFilter.java:154) > at > org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:143) > at > org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:122) > at > org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:96) > at > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) > at > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) > at > org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:157) > at > org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:136) > at > org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:84) > at > org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:293) > at > org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65) > at > org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1357) > at > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1214) > at > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1126) > at org.apache.cassandra.db.Table.getRow(Table.java:347) > at > org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:70) > at > org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1052) > INFO 19:20:51,397 Stop listening to thrift clients > > -- > Best Regards, > > Shammi Jayasinghe > Associate Tech Lead > WSO2, Inc.; http://wso2.com, > mobile: +94 71 4493085 > >
