[
https://issues.apache.org/jira/browse/CASSANDRA-10689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15021062#comment-15021062
]
Lior Chaga commented on CASSANDRA-10689:
----------------------------------------
Have similar issue with cassandra 2.1.11
While reading data have a lot of
org.apache.cassandra.io.compress.CorruptBlockException.
Thread count is rising drastically and g1gc fails to clear heap. Also,
out-traffic is much higher than on other nodes (about twice as much), I suspect
it's due to speculative retry but that just a wild guess.
Just before shutdown we get a lot of these:
{code}
WARN [epollEventLoopGroup-2-34] 2015-11-22 08:38:25,463 Slf4JLogger.java:151 -
Unexpected exception from an event executor:
java.lang.OutOfMemoryError: Java heap space
ERROR [MessagingService-Incoming-/10.10.1.206] 2015-11-22 08:38:25,463
CassandraDaemon.java:227 - Exception in thread
Thread[MessagingService-Incoming-/10.10.1.206,5,main]
java.lang.OutOfMemoryError: Java heap space
ERROR [MessagingService-Incoming-/172.16.124.194] 2015-11-22 08:38:25,464
CassandraDaemon.java:227 - Exception in thread
Thread[MessagingService-Incoming-/172.16.124.194,5,main]
java.lang.OutOfMemoryError: Java heap space
ERROR [MessagingService-Incoming-/10.10.1.216] 2015-11-22 08:38:25,464
CassandraDaemon.java:227 - Exception in thread
Thread[MessagingService-Incoming-/10.10.1.216,5,main]
java.lang.OutOfMemoryError: Java heap space
ERROR [MessagingService-Incoming-/172.16.124.232] 2015-11-22 08:38:25,464
CassandraDaemon.java:227 - Exception in thread
Thread[MessagingService-Incoming-/172.16.124.232,5,main]
java.lang.OutOfMemoryError: Java heap space
ERROR [MessagingService-Incoming-/172.16.124.228] 2015-11-22 08:38:25,464
CassandraDaemon.java:227 - Exception in thread
Thread[MessagingService-Incoming-/172.16.124.228,5,main]
java.lang.OutOfMemoryError: Java heap space
ERROR [MessagingService-Incoming-/10.10.1.225] 2015-11-22 08:38:25,464
CassandraDaemon.java:227 - Exception in thread
Thread[MessagingService-Incoming-/10.10.1.225,5,main]
java.lang.OutOfMemoryError: Java heap space
ERROR [MessagingService-Incoming-/172.16.124.203] 2015-11-22 08:38:25,464
CassandraDaemon.java:227 - Exception in thread
Thread[MessagingService-Incoming-/172.16.124.203,5,main]
java.lang.OutOfMemoryError: Java heap space
WARN [epollEventLoopGroup-2-47] 2015-11-22 08:38:25,464 Slf4JLogger.java:151 -
Unexpected exception in the selector loop.
java.lang.OutOfMemoryError: Java heap space
WARN [epollEventLoopGroup-2-5] 2015-11-22 08:38:25,464 Slf4JLogger.java:151 -
Unexpected exception in the selector loop.
{code}
and then these:
{code}
ERROR [Reference-Reaper:1] 2015-11-22 08:38:34,907 Ref.java:179 - LEAK
DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@7fd3d5e1) to class
org.apache.cassandra.io.sstable.SSTableReader$InstanceTidier@986793087:/var/lib/cassandra/data/disk1/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/system-sstable_activity-ka-4349
was not released before the reference was garbage collected
{code}
And there's also pending tasks:
{code}
- MemtableFlushWriter 6 22378 16046 0
0
- MemtablePostFlush 1 23620 17257 0
0
{code}
Haven't tried scrub yet.
> java.lang.OutOfMemoryError: Direct buffer memory
> ------------------------------------------------
>
> Key: CASSANDRA-10689
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10689
> Project: Cassandra
> Issue Type: Bug
> Reporter: mlowicki
>
> {code}
> ERROR [SharedPool-Worker-63] 2015-11-11 17:53:16,161
> JVMStabilityInspector.java:117 - JVM state determined to be unstable.
> Exiting forcefully due to:
> java.lang.OutOfMemoryError: Direct buffer memory
> at java.nio.Bits.reserveMemory(Bits.java:658) ~[na:1.7.0_80]
> at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
> ~[na:1.7.0_80]
> at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
> ~[na:1.7.0_80]
> at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:174)
> ~[na:1.7.0_80]
> at sun.nio.ch.IOUtil.read(IOUtil.java:195) ~[na:1.7.0_80]
> at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:149)
> ~[na:1.7.0_80]
> at
> org.apache.cassandra.io.compress.CompressedRandomAccessReader.decompressChunk(CompressedRandomAccessReader.java:104)
> ~[apache-cassandra-2.1.11.jar:2.1.11]
> at
> org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:81)
> ~[apache-cassandra-2.1.11.jar:2.1.11]
> at
> org.apache.cassandra.io.util.RandomAccessReader.seek(RandomAccessReader.java:310)
> ~[apache-cassandra-2.1.11.jar:2.1.11]
> at
> org.apache.cassandra.io.util.PoolingSegmentedFile.getSegment(PoolingSegmentedFile.java:64)
> ~[apache-cassandra-2.1.11.jar:2.1.11]
> at
> org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader.java:1894)
> ~[apache-cassandra-2.1.11.jar:2.1.11]
> at
> org.apache.cassandra.db.columniterator.IndexedSliceReader.setToRowStart(IndexedSliceReader.java:107)
> ~[apache-cassandra-2.1.11.jar:2.1.11]
> at
> org.apache.cassandra.db.columniterator.IndexedSliceReader.<init>(IndexedSliceReader.java:83)
> ~[apache-cassandra-2.1.11.jar:2.1.11]
> at
> org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:65)
> ~[apache-cassandra-2.1.11.jar:2.1.11]
> at
> org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:42)
> ~[apache-cassandra-2.1.11.jar:2.1.11]
> at
> org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:246)
> ~[apache-cassandra-2.1.11.jar:2.1.11]
> at
> org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:62)
> ~[apache-cassandra-2.1.11.jar:2.1.11]
> at
> org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:270)
> ~[apache-cassandra-2.1.11.jar:2.1.11]
> at
> org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:62)
> ~[apache-cassandra-2.1.11.jar:2.1.11]
> at
> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1994)
> ~[apache-cassandra-2.1.11.jar:2.1.11]
> at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1837)
> ~[apache-cassandra-2.1.11.jar:2.1.11]
> at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:353)
> ~[apache-cassandra-2.1.11.jar:2.1.11]
> at
> org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:85)
> ~[apache-cassandra-2.1.11.jar:2.1.11]
> at
> org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:47)
> ~[apache-cassandra-2.1.11.jar:2.1.11]
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
> ~[apache-cassandra-2.1.11.jar:2.1.11]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> ~[na:1.7.0_80]
> at
> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
> ~[apache-cassandra-2.1.11.jar:2.1.11]
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
> [apache-cassandra-2.1.11.jar:2.1.11]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_80]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
