Richard Hesse created CASSANDRA-16764:
-----------------------------------------
Summary: Compaction repeatedly fails validateReallocation exception
Key: CASSANDRA-16764
URL: https://issues.apache.org/jira/browse/CASSANDRA-16764
Project: Cassandra
Issue Type: Bug
Components: Local/Compaction/LCS
Reporter: Richard Hesse
I have a few nodes in my ring that are stuck repeatedly trying to compact the
same tables over and over again. I've run through the usual trick of rolling
restarts, and it doesn't seem to help. This exception is logged on the nodes:
{{ERROR [CompactionExecutor:6] 2021-06-25 20:28:30,001
CassandraDaemon.java:244 - Exception in thread
Thread[CompactionExecutor:6,1,main]
java.lang.RuntimeException: null
at
org.apache.cassandra.io.util.DataOutputBuffer.validateReallocation(DataOutputBuffer.java:134)
~[apache-cassandra-3.11.10.jar:3.11.10]
at
org.apache.cassandra.io.util.DataOutputBuffer.calculateNewSize(DataOutputBuffer.java:152)
~[apache-cassandra-3.11.10.jar:3.11.10]
at
org.apache.cassandra.io.util.DataOutputBuffer.expandToFit(DataOutputBuffer.java:159)
~[apache-cassandra-3.11.10.jar:3.11.10]
at
org.apache.cassandra.io.util.DataOutputBuffer.doFlush(DataOutputBuffer.java:119)
~[apache-cassandra-3.11.10.jar:3.11.10]
at
org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:132)
~[apache-cassandra-3.11.10.jar:3.11.10]
at
org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:151)
~[apache-cassandra-3.11.10.jar:3.11.10]
at
org.apache.cassandra.utils.ByteBufferUtil.writeWithVIntLength(ByteBufferUtil.java:296)
~[apache-cassandra-3.11.10.jar:3.11.10]
at
org.apache.cassandra.db.marshal.AbstractType.writeValue(AbstractType.java:426)
~[apache-cassandra-3.11.10.jar:3.11.10]
at
org.apache.cassandra.db.ClusteringPrefix$Serializer.serializeValuesWithoutSize(ClusteringPrefix.java:323)
~[apache-cassandra-3.11.10.jar:3.11.10]
at
org.apache.cassandra.db.Clustering$Serializer.serialize(Clustering.java:131)
~[apache-cassandra-3.11.10.jar:3.11.10]
at
org.apache.cassandra.db.ClusteringPrefix$Serializer.serialize(ClusteringPrefix.java:266)
~[apache-cassandra-3.11.10.jar:3.11.10]
at
org.apache.cassandra.db.Serializers$NewFormatSerializer.serialize(Serializers.java:167)
~[apache-cassandra-3.11.10.jar:3.11.10]
at
org.apache.cassandra.db.Serializers$NewFormatSerializer.serialize(Serializers.java:154)
~[apache-cassandra-3.11.10.jar:3.11.10]
at
org.apache.cassandra.io.sstable.IndexInfo$Serializer.serialize(IndexInfo.java:103)
~[apache-cassandra-3.11.10.jar:3.11.10]
at
org.apache.cassandra.io.sstable.IndexInfo$Serializer.serialize(IndexInfo.java:82)
~[apache-cassandra-3.11.10.jar:3.11.10]
at
org.apache.cassandra.db.ColumnIndex.addIndexBlock(ColumnIndex.java:216)
~[apache-cassandra-3.11.10.jar:3.11.10]
at org.apache.cassandra.db.ColumnIndex.add(ColumnIndex.java:264)
~[apache-cassandra-3.11.10.jar:3.11.10]
at
org.apache.cassandra.db.ColumnIndex.buildRowIndex(ColumnIndex.java:111)
~[apache-cassandra-3.11.10.jar:3.11.10]
at
org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:173)
~[apache-cassandra-3.11.10.jar:3.11.10]
at
org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:136)
~[apache-cassandra-3.11.10.jar:3.11.10]
at
org.apache.cassandra.db.compaction.writers.MaxSSTableSizeWriter.realAppend(MaxSSTableSizeWriter.java:98)
~[apache-cassandra-3.11.10.jar:3.11.10]
at
org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:143)
~[apache-cassandra-3.11.10.jar:3.11.10]
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:204)
~[apache-cassandra-3.11.10.jar:3.11.10]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[apache-cassandra-3.11.10.jar:3.11.10]
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85)
~[apache-cassandra-3.11.10.jar:3.11.10]
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
~[apache-cassandra-3.11.10.jar:3.11.10]
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:272)
~[apache-cassandra-3.11.10.jar:3.11.10]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_292]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[na:1.8.0_292]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
~[na:1.8.0_292]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[na:1.8.0_292]
at
org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84)
[apache-cassandra-3.11.10.jar:3.11.10]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_292]}}
Watching the compaction progress, it looks like it makes it through, dies, then
starts over again. This process repeats forever.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
