[
https://issues.apache.org/jira/browse/CASSANDRA-8851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14362889#comment-14362889
]
Tobias Schlottke commented on CASSANDRA-8851:
---------------------------------------------
Hey [~benedict]!
I recently noticed that even with your patch, there are still some exceptions
of that kind happening. Example:
{code}
ERROR [CompactionExecutor:3] 2015-03-16 08:34:24,798 CassandraDaemon.java:167 -
Exception in thread Thread[CompactionExecutor:3,1,main]
java.lang.NullPointerException: null
at
org.apache.cassandra.io.sstable.SSTableReader.cloneWithNewStart(SSTableReader.java:892)
~[apache-cassandra-2.1.3.jar:2.1.3]
at
org.apache.cassandra.io.sstable.SSTableRewriter.moveStarts(SSTableRewriter.java:328)
~[apache-cassandra-2.1.3.jar:2.1.3]
at
org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:178)
~[apache-cassandra-2.1.3.jar:2.1.3]
at
org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:117)
~[apache-cassandra-2.1.3.jar:2.1.3]
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:192)
~[apache-cassandra-2.1.3.jar:2.1.3]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[apache-cassandra-2.1.3.jar:2.1.3]
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:72)
~[apache-cassandra-2.1.3.jar:2.1.3]
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
~[apache-cassandra-2.1.3.jar:2.1.3]
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:240)
~[apache-cassandra-2.1.3.jar:2.1.3]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
~[na:1.7.0_45]
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
~[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_45]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
ERROR [Reference-Reaper:1] 2015-03-16 08:34:27,210 Ref.java:181 - LEAK
DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@24b4e762) to class
org.apache.cassandra.io.sstable.SSTableReader$InstanceTidier@1623385537:/data/cassandra/data/metrigo_prod/new_user_data/metrigo_prod-new_user_data-jb-164099
was not released before the reference was garbage collected
{code}
Any ideas?
Best,
Tobias
> Uncaught exception on thread Thread[SharedPool-Worker-16,5,main] after
> upgrade to 2.1.3
> ---------------------------------------------------------------------------------------
>
> Key: CASSANDRA-8851
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8851
> Project: Cassandra
> Issue Type: Bug
> Environment: ubuntu
> Reporter: Tobias Schlottke
> Assignee: Benedict
> Priority: Critical
> Fix For: 2.1.4
>
>
> Hi there,
> after upgrading to 2.1.3 we've got the following error every few seconds:
> {code}
> WARN [SharedPool-Worker-16] 2015-02-23 10:20:36,392
> AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread
> Thread[SharedPool-Worker-16,5,main]: {}
> java.lang.AssertionError: null
> at org.apache.cassandra.io.util.Memory.size(Memory.java:307)
> ~[apache-cassandra-2.1.3.jar:2.1.3]
> at
> org.apache.cassandra.utils.obs.OffHeapBitSet.capacity(OffHeapBitSet.java:61)
> ~[apache-cassandra-2.1.3.jar:2.1.3]
> at org.apache.cassandra.utils.BloomFilter.indexes(BloomFilter.java:74)
> ~[apache-cassandra-2.1.3.jar:2.1.3]
> at
> org.apache.cassandra.utils.BloomFilter.isPresent(BloomFilter.java:98)
> ~[apache-cassandra-2.1.3.jar:2.1.3]
> at
> org.apache.cassandra.io.sstable.SSTableReader.getPosition(SSTableReader.java:1366)
> ~[apache-cassandra-2.1.3.jar:2.1.3]
> at
> org.apache.cassandra.io.sstable.SSTableReader.getPosition(SSTableReader.java:1350)
> ~[apache-cassandra-2.1.3.jar:2.1.3]
> at
> org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:41)
> ~[apache-cassandra-2.1.3.jar:2.1.3]
> at
> org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:185)
> ~[apache-cassandra-2.1.3.jar:2.1.3]
> at
> org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:62)
> ~[apache-cassandra-2.1.3.jar:2.1.3]
> at
> org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:273)
> ~[apache-cassandra-2.1.3.jar:2.1.3]
> at
> org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:62)
> ~[apache-cassandra-2.1.3.jar:2.1.3]
> at
> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1915)
> ~[apache-cassandra-2.1.3.jar:2.1.3]
> at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1748)
> ~[apache-cassandra-2.1.3.jar:2.1.3]
> at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:342)
> ~[apache-cassandra-2.1.3.jar:2.1.3]
> at
> org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:57)
> ~[apache-cassandra-2.1.3.jar:2.1.3]
> at
> org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:47)
> ~[apache-cassandra-2.1.3.jar:2.1.3]
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62)
> ~[apache-cassandra-2.1.3.jar:2.1.3]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> ~[na:1.7.0_45]
> at
> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
> ~[apache-cassandra-2.1.3.jar:2.1.3]
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
> [apache-cassandra-2.1.3.jar:2.1.3]
> at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
> {code}
> This seems to crash the compactions and pushes up server load and piles up
> compactions.
> Any idea / possible workaround?
> Best,
> Tobias
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
