[ https://issues.apache.org/jira/browse/CASSANDRA-10615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14979504#comment-14979504 ]
Jeff Jirsa commented on CASSANDRA-10615: ---------------------------------------- JVM OOM was one of the canonical risks of enabling the row cache older versions of C* before the row cache was made more like a query cache in CASSANDRA-5357. In 2.1 and newer, options have been added to control how many rows per partition could be cached ( {{rows_per_partition_to_cache}} ) - were/are you using the defaults? Do you have the serialized cache files saved? > Large partition plus row cache causes JVM OutOfMemory Error > ----------------------------------------------------------- > > Key: CASSANDRA-10615 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10615 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: Cassandra 2.1.8.621 > Reporter: Michael Perrone > > Found on multiple nodes of cluster, restarting C* resulted in hung startup > process, nodes never came up to Up Joining, but would hang until OOM. The > work-around is to disable row cache. > {code} > ERROR [SharedPool-Worker-22] 2015-10-27 17:22:52,753 > JVMStabilityInspector.java:94 - JVM state determined to be unstable. Exiting > forcefully due to: > java.lang.OutOfMemoryError: Java heap space > at > org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:374) > ~[cassandra-all-2.1.8.621.jar:2.1.8.621] > at > org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:348) > ~[cassandra-all-2.1.8.621.jar:2.1.8.621] > at > org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:327) > ~[cassandra-all-2.1.8.621.jar:2.1.8.621] > at > org.apache.cassandra.db.composites.AbstractCType$Serializer.deserialize(AbstractCType.java:397) > ~[cassandra-all-2.1.8.621.jar:2.1.8.621] > at > org.apache.cassandra.db.composites.AbstractCType$Serializer.deserialize(AbstractCType.java:381) > ~[cassandra-all-2.1.8.621.jar:2.1.8.621] > at > org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:75) > ~[cassandra-all-2.1.8.621.jar:2.1.8.621] > at > org.apache.cassandra.db.AbstractCell$1.computeNext(AbstractCell.java:52) > ~[cassandra-all-2.1.8.621.jar:2.1.8.621] > at > org.apache.cassandra.db.AbstractCell$1.computeNext(AbstractCell.java:46) > ~[cassandra-all-2.1.8.621.jar:2.1.8.621] > at > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) > ~[guava-16.0.1.jar:na] > at > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) > ~[guava-16.0.1.jar:na] > at > org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:83) > ~[cassandra-all-2.1.8.621.jar:2.1.8.621] > at > org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:37) > ~[cassandra-all-2.1.8.621.jar:2.1.8.621] > at > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) > ~[guava-16.0.1.jar:na] > at > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) > ~[guava-16.0.1.jar:na] > at > org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:82) > ~[cassandra-all-2.1.8.621.jar:2.1.8.621] > at > org.apache.cassandra.db.filter.QueryFilter$2.getNext(QueryFilter.java:173) > ~[cassandra-all-2.1.8.621.jar:2.1.8.621] > at > org.apache.cassandra.db.filter.QueryFilter$2.hasNext(QueryFilter.java:156) > ~[cassandra-all-2.1.8.621.jar:2.1.8.621] > at > org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:146) > ~[cassandra-all-2.1.8.621.jar:2.1.8.621] > at > org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:125) > ~[cassandra-all-2.1.8.621.jar:2.1.8.621] > at > org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:99) > ~[cassandra-all-2.1.8.621.jar:2.1.8.621] > at > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) > ~[guava-16.0.1.jar:na] > at > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) > ~[guava-16.0.1.jar:na] > at > org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:264) > ~[cassandra-all-2.1.8.621.jar:2.1.8.621] > at > org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:108) > ~[cassandra-all-2.1.8.621.jar:2.1.8.621] > at > org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:82) > ~[cassandra-all-2.1.8.621.jar:2.1.8.621] > at > org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:69) > ~[cassandra-all-2.1.8.621.jar:2.1.8.621] > at > org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:314) > ~[cassandra-all-2.1.8.621.jar:2.1.8.621] > at > org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:62) > ~[cassandra-all-2.1.8.621.jar:2.1.8.621] > at > org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1967) > ~[cassandra-all-2.1.8.621.jar:2.1.8.621] > at > org.apache.cassandra.service.CacheService$RowCacheSerializer$1.call(CacheService.java:456) > ~[cassandra-all-2.1.8.621.jar:2.1.8.621] > at > org.apache.cassandra.service.CacheService$RowCacheSerializer$1.call(CacheService.java:451) > ~[cassandra-all-2.1.8.621.jar:2.1.8.621] > at > org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) > ~[cassandra-all-2.1.8.621.jar:2.1.8.621] > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)