[jira] [Commented] (CASSANDRA-16764) Compaction repeatedly fails validateReallocation exception
[
https://issues.apache.org/jira/browse/CASSANDRA-16764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17513546#comment-17513546
]
Jai Bheemsen Rao Dhanwada commented on CASSANDRA-16764:
---
I am running into this issue as well and have couple of questions.
# I understand that Index file is `SSTable Index which maps row keys to their
respective offsets in the Data file`, but does this holds the actual Row Key
and since the compaction is failing, even the tombstoned rows are not getting
cleared from the Index files and the actual data files (Data.db) ?
# Is there a configuration/limit where we can increase this limit from 2GB to
a higher value?
# In worst case, what is the harm or loss when we delete these files?
> Compaction repeatedly fails validateReallocation exception
> --
>
> Key: CASSANDRA-16764
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16764
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Compaction/LCS
>Reporter: Richard Hesse
>Priority: Normal
>
> I have a few nodes in my ring that are stuck repeatedly trying to compact the
> same tables over and over again. I've run through the usual trick of rolling
> restarts, and it doesn't seem to help. This exception is logged on the nodes:
> {code}
> ERROR [CompactionExecutor:6] 2021-06-25 20:28:30,001 CassandraDaemon.java:244
> - Exception in thread Thread[CompactionExecutor:6,1,main]
> java.lang.RuntimeException: null
> at
> org.apache.cassandra.io.util.DataOutputBuffer.validateReallocation(DataOutputBuffer.java:134)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.calculateNewSize(DataOutputBuffer.java:152)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.expandToFit(DataOutputBuffer.java:159)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.doFlush(DataOutputBuffer.java:119)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:132)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:151)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.utils.ByteBufferUtil.writeWithVIntLength(ByteBufferUtil.java:296)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.marshal.AbstractType.writeValue(AbstractType.java:426)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ClusteringPrefix$Serializer.serializeValuesWithoutSize(ClusteringPrefix.java:323)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Clustering$Serializer.serialize(Clustering.java:131)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ClusteringPrefix$Serializer.serialize(ClusteringPrefix.java:266)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Serializers$NewFormatSerializer.serialize(Serializers.java:167)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Serializers$NewFormatSerializer.serialize(Serializers.java:154)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.IndexInfo$Serializer.serialize(IndexInfo.java:103)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.IndexInfo$Serializer.serialize(IndexInfo.java:82)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ColumnIndex.addIndexBlock(ColumnIndex.java:216)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.db.ColumnIndex.add(ColumnIndex.java:264)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ColumnIndex.buildRowIndex(ColumnIndex.java:111)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:173)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:136)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.writers.MaxSSTableSizeWriter.realAppend(MaxSSTableSizeWriter.java:98)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:143)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:204)
>
[jira] [Commented] (CASSANDRA-16764) Compaction repeatedly fails validateReallocation exception
[
https://issues.apache.org/jira/browse/CASSANDRA-16764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17396746#comment-17396746
]
Richard Hesse commented on CASSANDRA-16764:
---
Updating the bug. We had some success moving those data files related to large
(~1.9GB) index files out of the way. Those are the large files around that max
file size. Still playing whack-a-mole as repairs can finally run and more
compactions roll through and create new, up-to-2GB index files. But overall I
think we know our path forward. Thanks for everyone's help!
> Compaction repeatedly fails validateReallocation exception
> --
>
> Key: CASSANDRA-16764
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16764
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Compaction/LCS
>Reporter: Richard Hesse
>Priority: Normal
>
> I have a few nodes in my ring that are stuck repeatedly trying to compact the
> same tables over and over again. I've run through the usual trick of rolling
> restarts, and it doesn't seem to help. This exception is logged on the nodes:
> {code}
> ERROR [CompactionExecutor:6] 2021-06-25 20:28:30,001 CassandraDaemon.java:244
> - Exception in thread Thread[CompactionExecutor:6,1,main]
> java.lang.RuntimeException: null
> at
> org.apache.cassandra.io.util.DataOutputBuffer.validateReallocation(DataOutputBuffer.java:134)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.calculateNewSize(DataOutputBuffer.java:152)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.expandToFit(DataOutputBuffer.java:159)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.doFlush(DataOutputBuffer.java:119)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:132)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:151)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.utils.ByteBufferUtil.writeWithVIntLength(ByteBufferUtil.java:296)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.marshal.AbstractType.writeValue(AbstractType.java:426)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ClusteringPrefix$Serializer.serializeValuesWithoutSize(ClusteringPrefix.java:323)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Clustering$Serializer.serialize(Clustering.java:131)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ClusteringPrefix$Serializer.serialize(ClusteringPrefix.java:266)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Serializers$NewFormatSerializer.serialize(Serializers.java:167)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Serializers$NewFormatSerializer.serialize(Serializers.java:154)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.IndexInfo$Serializer.serialize(IndexInfo.java:103)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.IndexInfo$Serializer.serialize(IndexInfo.java:82)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ColumnIndex.addIndexBlock(ColumnIndex.java:216)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.db.ColumnIndex.add(ColumnIndex.java:264)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ColumnIndex.buildRowIndex(ColumnIndex.java:111)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:173)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:136)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.writers.MaxSSTableSizeWriter.realAppend(MaxSSTableSizeWriter.java:98)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:143)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:204)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
>
[jira] [Commented] (CASSANDRA-16764) Compaction repeatedly fails validateReallocation exception
[
https://issues.apache.org/jira/browse/CASSANDRA-16764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385457#comment-17385457
]
Brandon Williams commented on CASSANDRA-16764:
--
That's correct. I would probably just move the files out of the way first
though so you can put them back if something goes wrong.
> Compaction repeatedly fails validateReallocation exception
> --
>
> Key: CASSANDRA-16764
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16764
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Compaction/LCS
>Reporter: Richard Hesse
>Priority: Normal
>
> I have a few nodes in my ring that are stuck repeatedly trying to compact the
> same tables over and over again. I've run through the usual trick of rolling
> restarts, and it doesn't seem to help. This exception is logged on the nodes:
> {code}
> ERROR [CompactionExecutor:6] 2021-06-25 20:28:30,001 CassandraDaemon.java:244
> - Exception in thread Thread[CompactionExecutor:6,1,main]
> java.lang.RuntimeException: null
> at
> org.apache.cassandra.io.util.DataOutputBuffer.validateReallocation(DataOutputBuffer.java:134)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.calculateNewSize(DataOutputBuffer.java:152)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.expandToFit(DataOutputBuffer.java:159)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.doFlush(DataOutputBuffer.java:119)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:132)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:151)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.utils.ByteBufferUtil.writeWithVIntLength(ByteBufferUtil.java:296)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.marshal.AbstractType.writeValue(AbstractType.java:426)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ClusteringPrefix$Serializer.serializeValuesWithoutSize(ClusteringPrefix.java:323)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Clustering$Serializer.serialize(Clustering.java:131)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ClusteringPrefix$Serializer.serialize(ClusteringPrefix.java:266)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Serializers$NewFormatSerializer.serialize(Serializers.java:167)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Serializers$NewFormatSerializer.serialize(Serializers.java:154)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.IndexInfo$Serializer.serialize(IndexInfo.java:103)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.IndexInfo$Serializer.serialize(IndexInfo.java:82)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ColumnIndex.addIndexBlock(ColumnIndex.java:216)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.db.ColumnIndex.add(ColumnIndex.java:264)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ColumnIndex.buildRowIndex(ColumnIndex.java:111)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:173)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:136)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.writers.MaxSSTableSizeWriter.realAppend(MaxSSTableSizeWriter.java:98)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:143)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:204)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
>
[jira] [Commented] (CASSANDRA-16764) Compaction repeatedly fails validateReallocation exception
[
https://issues.apache.org/jira/browse/CASSANDRA-16764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385189#comment-17385189
]
Richard Hesse commented on CASSANDRA-16764:
---
Just to verify, the procedure there is: stop Cassandra, delete all files
referencing that named sstable, start Cassandra (repeat on all primary range
nodes in the ring)? I've got a list of the nodes and files; they're pretty easy
to spot based on compactions outstanding. I just wanted to double check the
procedure. I've never actually tried to willfully lose data in Cassandra
before.
> Compaction repeatedly fails validateReallocation exception
> --
>
> Key: CASSANDRA-16764
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16764
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Compaction/LCS
>Reporter: Richard Hesse
>Priority: Normal
>
> I have a few nodes in my ring that are stuck repeatedly trying to compact the
> same tables over and over again. I've run through the usual trick of rolling
> restarts, and it doesn't seem to help. This exception is logged on the nodes:
> {code}
> ERROR [CompactionExecutor:6] 2021-06-25 20:28:30,001 CassandraDaemon.java:244
> - Exception in thread Thread[CompactionExecutor:6,1,main]
> java.lang.RuntimeException: null
> at
> org.apache.cassandra.io.util.DataOutputBuffer.validateReallocation(DataOutputBuffer.java:134)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.calculateNewSize(DataOutputBuffer.java:152)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.expandToFit(DataOutputBuffer.java:159)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.doFlush(DataOutputBuffer.java:119)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:132)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:151)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.utils.ByteBufferUtil.writeWithVIntLength(ByteBufferUtil.java:296)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.marshal.AbstractType.writeValue(AbstractType.java:426)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ClusteringPrefix$Serializer.serializeValuesWithoutSize(ClusteringPrefix.java:323)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Clustering$Serializer.serialize(Clustering.java:131)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ClusteringPrefix$Serializer.serialize(ClusteringPrefix.java:266)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Serializers$NewFormatSerializer.serialize(Serializers.java:167)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Serializers$NewFormatSerializer.serialize(Serializers.java:154)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.IndexInfo$Serializer.serialize(IndexInfo.java:103)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.IndexInfo$Serializer.serialize(IndexInfo.java:82)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ColumnIndex.addIndexBlock(ColumnIndex.java:216)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.db.ColumnIndex.add(ColumnIndex.java:264)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ColumnIndex.buildRowIndex(ColumnIndex.java:111)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:173)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:136)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.writers.MaxSSTableSizeWriter.realAppend(MaxSSTableSizeWriter.java:98)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:143)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:204)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
>
[jira] [Commented] (CASSANDRA-16764) Compaction repeatedly fails validateReallocation exception
[
https://issues.apache.org/jira/browse/CASSANDRA-16764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17383320#comment-17383320
]
Brandon Williams commented on CASSANDRA-16764:
--
If that is an option that is probably the simplest thing to do.
> Compaction repeatedly fails validateReallocation exception
> --
>
> Key: CASSANDRA-16764
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16764
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Compaction/LCS
>Reporter: Richard Hesse
>Priority: Normal
>
> I have a few nodes in my ring that are stuck repeatedly trying to compact the
> same tables over and over again. I've run through the usual trick of rolling
> restarts, and it doesn't seem to help. This exception is logged on the nodes:
> {code}
> ERROR [CompactionExecutor:6] 2021-06-25 20:28:30,001 CassandraDaemon.java:244
> - Exception in thread Thread[CompactionExecutor:6,1,main]
> java.lang.RuntimeException: null
> at
> org.apache.cassandra.io.util.DataOutputBuffer.validateReallocation(DataOutputBuffer.java:134)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.calculateNewSize(DataOutputBuffer.java:152)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.expandToFit(DataOutputBuffer.java:159)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.doFlush(DataOutputBuffer.java:119)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:132)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:151)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.utils.ByteBufferUtil.writeWithVIntLength(ByteBufferUtil.java:296)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.marshal.AbstractType.writeValue(AbstractType.java:426)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ClusteringPrefix$Serializer.serializeValuesWithoutSize(ClusteringPrefix.java:323)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Clustering$Serializer.serialize(Clustering.java:131)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ClusteringPrefix$Serializer.serialize(ClusteringPrefix.java:266)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Serializers$NewFormatSerializer.serialize(Serializers.java:167)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Serializers$NewFormatSerializer.serialize(Serializers.java:154)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.IndexInfo$Serializer.serialize(IndexInfo.java:103)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.IndexInfo$Serializer.serialize(IndexInfo.java:82)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ColumnIndex.addIndexBlock(ColumnIndex.java:216)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.db.ColumnIndex.add(ColumnIndex.java:264)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ColumnIndex.buildRowIndex(ColumnIndex.java:111)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:173)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:136)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.writers.MaxSSTableSizeWriter.realAppend(MaxSSTableSizeWriter.java:98)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:143)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:204)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
>
[jira] [Commented] (CASSANDRA-16764) Compaction repeatedly fails validateReallocation exception
[
https://issues.apache.org/jira/browse/CASSANDRA-16764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17382359#comment-17382359
]
Richard Hesse commented on CASSANDRA-16764:
---
Yes, it's LCS. The SSTable data file has a matching huge index file, which
makes sense given entry size. Other than that nothing really stands out. It
looks like our only path forward here is to delete the file and reload the data
minus the bad entry?
> Compaction repeatedly fails validateReallocation exception
> --
>
> Key: CASSANDRA-16764
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16764
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Compaction/LCS
>Reporter: Richard Hesse
>Priority: Normal
>
> I have a few nodes in my ring that are stuck repeatedly trying to compact the
> same tables over and over again. I've run through the usual trick of rolling
> restarts, and it doesn't seem to help. This exception is logged on the nodes:
> {code}
> ERROR [CompactionExecutor:6] 2021-06-25 20:28:30,001 CassandraDaemon.java:244
> - Exception in thread Thread[CompactionExecutor:6,1,main]
> java.lang.RuntimeException: null
> at
> org.apache.cassandra.io.util.DataOutputBuffer.validateReallocation(DataOutputBuffer.java:134)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.calculateNewSize(DataOutputBuffer.java:152)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.expandToFit(DataOutputBuffer.java:159)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.doFlush(DataOutputBuffer.java:119)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:132)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:151)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.utils.ByteBufferUtil.writeWithVIntLength(ByteBufferUtil.java:296)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.marshal.AbstractType.writeValue(AbstractType.java:426)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ClusteringPrefix$Serializer.serializeValuesWithoutSize(ClusteringPrefix.java:323)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Clustering$Serializer.serialize(Clustering.java:131)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ClusteringPrefix$Serializer.serialize(ClusteringPrefix.java:266)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Serializers$NewFormatSerializer.serialize(Serializers.java:167)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Serializers$NewFormatSerializer.serialize(Serializers.java:154)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.IndexInfo$Serializer.serialize(IndexInfo.java:103)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.IndexInfo$Serializer.serialize(IndexInfo.java:82)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ColumnIndex.addIndexBlock(ColumnIndex.java:216)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.db.ColumnIndex.add(ColumnIndex.java:264)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ColumnIndex.buildRowIndex(ColumnIndex.java:111)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:173)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:136)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.writers.MaxSSTableSizeWriter.realAppend(MaxSSTableSizeWriter.java:98)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:143)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:204)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
>
[jira] [Commented] (CASSANDRA-16764) Compaction repeatedly fails validateReallocation exception
[
https://issues.apache.org/jira/browse/CASSANDRA-16764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17382323#comment-17382323
]
Brandon Williams commented on CASSANDRA-16764:
--
Is this table using LCS? This would make sense if so, since it defaults to
writing 160MB sstables, but also does not split rows, so this may already be
the sstable we're after. Does that large partition key line up with any other
findings?
> Compaction repeatedly fails validateReallocation exception
> --
>
> Key: CASSANDRA-16764
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16764
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Compaction/LCS
>Reporter: Richard Hesse
>Priority: Normal
>
> I have a few nodes in my ring that are stuck repeatedly trying to compact the
> same tables over and over again. I've run through the usual trick of rolling
> restarts, and it doesn't seem to help. This exception is logged on the nodes:
> {code}
> ERROR [CompactionExecutor:6] 2021-06-25 20:28:30,001 CassandraDaemon.java:244
> - Exception in thread Thread[CompactionExecutor:6,1,main]
> java.lang.RuntimeException: null
> at
> org.apache.cassandra.io.util.DataOutputBuffer.validateReallocation(DataOutputBuffer.java:134)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.calculateNewSize(DataOutputBuffer.java:152)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.expandToFit(DataOutputBuffer.java:159)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.doFlush(DataOutputBuffer.java:119)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:132)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:151)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.utils.ByteBufferUtil.writeWithVIntLength(ByteBufferUtil.java:296)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.marshal.AbstractType.writeValue(AbstractType.java:426)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ClusteringPrefix$Serializer.serializeValuesWithoutSize(ClusteringPrefix.java:323)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Clustering$Serializer.serialize(Clustering.java:131)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ClusteringPrefix$Serializer.serialize(ClusteringPrefix.java:266)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Serializers$NewFormatSerializer.serialize(Serializers.java:167)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Serializers$NewFormatSerializer.serialize(Serializers.java:154)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.IndexInfo$Serializer.serialize(IndexInfo.java:103)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.IndexInfo$Serializer.serialize(IndexInfo.java:82)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ColumnIndex.addIndexBlock(ColumnIndex.java:216)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.db.ColumnIndex.add(ColumnIndex.java:264)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ColumnIndex.buildRowIndex(ColumnIndex.java:111)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:173)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:136)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.writers.MaxSSTableSizeWriter.realAppend(MaxSSTableSizeWriter.java:98)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:143)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:204)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
[jira] [Commented] (CASSANDRA-16764) Compaction repeatedly fails validateReallocation exception
[
https://issues.apache.org/jira/browse/CASSANDRA-16764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17382317#comment-17382317
]
Richard Hesse commented on CASSANDRA-16764:
---
Scrub can detect the length though, and throw out the bad data. That's what I
was alluding to. Back to our splitting approach, it doesn't seem to actually do
anything. I experimented with various sizes without effect. It creates a new
file of same size and deletes the original. Example output:
{noformat}
$ sstablesplit --no-snapshot -s 2000 --debug md-93768-big-Data.db
WARN 20:10:33 Live sstable md-93768-big-Data.db from level 0 is not on
corresponding level in the leveled manifest. This is not a problem per se, but
may indicate an orphaned sstable due to a failed compaction not cleaned up
properly.
WARN 20:14:30 Writing large partition
pages/publish_data:7881957:200995803593712285 (43.027GiB) to sstable
md-93769-big-Data.db
{noformat}
> Compaction repeatedly fails validateReallocation exception
> --
>
> Key: CASSANDRA-16764
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16764
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Compaction/LCS
>Reporter: Richard Hesse
>Priority: Normal
>
> I have a few nodes in my ring that are stuck repeatedly trying to compact the
> same tables over and over again. I've run through the usual trick of rolling
> restarts, and it doesn't seem to help. This exception is logged on the nodes:
> {code}
> ERROR [CompactionExecutor:6] 2021-06-25 20:28:30,001 CassandraDaemon.java:244
> - Exception in thread Thread[CompactionExecutor:6,1,main]
> java.lang.RuntimeException: null
> at
> org.apache.cassandra.io.util.DataOutputBuffer.validateReallocation(DataOutputBuffer.java:134)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.calculateNewSize(DataOutputBuffer.java:152)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.expandToFit(DataOutputBuffer.java:159)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.doFlush(DataOutputBuffer.java:119)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:132)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:151)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.utils.ByteBufferUtil.writeWithVIntLength(ByteBufferUtil.java:296)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.marshal.AbstractType.writeValue(AbstractType.java:426)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ClusteringPrefix$Serializer.serializeValuesWithoutSize(ClusteringPrefix.java:323)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Clustering$Serializer.serialize(Clustering.java:131)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ClusteringPrefix$Serializer.serialize(ClusteringPrefix.java:266)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Serializers$NewFormatSerializer.serialize(Serializers.java:167)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Serializers$NewFormatSerializer.serialize(Serializers.java:154)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.IndexInfo$Serializer.serialize(IndexInfo.java:103)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.IndexInfo$Serializer.serialize(IndexInfo.java:82)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ColumnIndex.addIndexBlock(ColumnIndex.java:216)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.db.ColumnIndex.add(ColumnIndex.java:264)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ColumnIndex.buildRowIndex(ColumnIndex.java:111)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:173)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:136)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.writers.MaxSSTableSizeWriter.realAppend(MaxSSTableSizeWriter.java:98)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
>
[jira] [Commented] (CASSANDRA-16764) Compaction repeatedly fails validateReallocation exception
[
https://issues.apache.org/jira/browse/CASSANDRA-16764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17382258#comment-17382258
]
Brandon Williams commented on CASSANDRA-16764:
--
This is probably just from sstablesplit having a default heap of 256M. If you
set MAX_HEAP_SIZE to something more reasonable that may work.
bq. If Cassandra can't compact
Scrub doesn't compact, so it can't know that.
> Compaction repeatedly fails validateReallocation exception
> --
>
> Key: CASSANDRA-16764
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16764
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Compaction/LCS
>Reporter: Richard Hesse
>Priority: Normal
>
> I have a few nodes in my ring that are stuck repeatedly trying to compact the
> same tables over and over again. I've run through the usual trick of rolling
> restarts, and it doesn't seem to help. This exception is logged on the nodes:
> {code}
> ERROR [CompactionExecutor:6] 2021-06-25 20:28:30,001 CassandraDaemon.java:244
> - Exception in thread Thread[CompactionExecutor:6,1,main]
> java.lang.RuntimeException: null
> at
> org.apache.cassandra.io.util.DataOutputBuffer.validateReallocation(DataOutputBuffer.java:134)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.calculateNewSize(DataOutputBuffer.java:152)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.expandToFit(DataOutputBuffer.java:159)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.doFlush(DataOutputBuffer.java:119)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:132)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:151)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.utils.ByteBufferUtil.writeWithVIntLength(ByteBufferUtil.java:296)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.marshal.AbstractType.writeValue(AbstractType.java:426)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ClusteringPrefix$Serializer.serializeValuesWithoutSize(ClusteringPrefix.java:323)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Clustering$Serializer.serialize(Clustering.java:131)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ClusteringPrefix$Serializer.serialize(ClusteringPrefix.java:266)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Serializers$NewFormatSerializer.serialize(Serializers.java:167)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Serializers$NewFormatSerializer.serialize(Serializers.java:154)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.IndexInfo$Serializer.serialize(IndexInfo.java:103)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.IndexInfo$Serializer.serialize(IndexInfo.java:82)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ColumnIndex.addIndexBlock(ColumnIndex.java:216)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.db.ColumnIndex.add(ColumnIndex.java:264)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ColumnIndex.buildRowIndex(ColumnIndex.java:111)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:173)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:136)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.writers.MaxSSTableSizeWriter.realAppend(MaxSSTableSizeWriter.java:98)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:143)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:204)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
>
[jira] [Commented] (CASSANDRA-16764) Compaction repeatedly fails validateReallocation exception
[
https://issues.apache.org/jira/browse/CASSANDRA-16764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17382253#comment-17382253
]
Richard Hesse commented on CASSANDRA-16764:
---
It was a good try, but no joy. We're getting the original heap error, but this
time on the splitting side:
{noformat}
$ sstablesplit --size 1700 --no-snapshot md-5357-big-Data.db
WARN 17:55:50 Live sstable {REDACTED}/md-5357-big-Data.db from level 3 is not
on corresponding level in the leveled manifest. This is not a problem per se,
but may indicate an orphaned sstable due to a failed compaction not cleaned up
properly.
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.(HeapByteBuffer.java:57)
at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
at
org.apache.cassandra.io.util.DataOutputBuffer.expandToFit(DataOutputBuffer.java:159)
at
org.apache.cassandra.io.util.DataOutputBuffer.doFlush(DataOutputBuffer.java:119)
at
org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:132)
at
org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:151)
...
{noformat}
Going back to my scrub comment from earlier, it might be worth considering. If
Cassandra can't compact or split the data, it's probably worth considering that
data corrupt from an operational standpoint.
> Compaction repeatedly fails validateReallocation exception
> --
>
> Key: CASSANDRA-16764
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16764
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Compaction/LCS
>Reporter: Richard Hesse
>Priority: Normal
>
> I have a few nodes in my ring that are stuck repeatedly trying to compact the
> same tables over and over again. I've run through the usual trick of rolling
> restarts, and it doesn't seem to help. This exception is logged on the nodes:
> {code}
> ERROR [CompactionExecutor:6] 2021-06-25 20:28:30,001 CassandraDaemon.java:244
> - Exception in thread Thread[CompactionExecutor:6,1,main]
> java.lang.RuntimeException: null
> at
> org.apache.cassandra.io.util.DataOutputBuffer.validateReallocation(DataOutputBuffer.java:134)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.calculateNewSize(DataOutputBuffer.java:152)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.expandToFit(DataOutputBuffer.java:159)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.doFlush(DataOutputBuffer.java:119)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:132)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:151)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.utils.ByteBufferUtil.writeWithVIntLength(ByteBufferUtil.java:296)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.marshal.AbstractType.writeValue(AbstractType.java:426)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ClusteringPrefix$Serializer.serializeValuesWithoutSize(ClusteringPrefix.java:323)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Clustering$Serializer.serialize(Clustering.java:131)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ClusteringPrefix$Serializer.serialize(ClusteringPrefix.java:266)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Serializers$NewFormatSerializer.serialize(Serializers.java:167)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Serializers$NewFormatSerializer.serialize(Serializers.java:154)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.IndexInfo$Serializer.serialize(IndexInfo.java:103)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.IndexInfo$Serializer.serialize(IndexInfo.java:82)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ColumnIndex.addIndexBlock(ColumnIndex.java:216)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.db.ColumnIndex.add(ColumnIndex.java:264)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ColumnIndex.buildRowIndex(ColumnIndex.java:111)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
>
[jira] [Commented] (CASSANDRA-16764) Compaction repeatedly fails validateReallocation exception
[
https://issues.apache.org/jira/browse/CASSANDRA-16764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17382150#comment-17382150
]
Richard Hesse commented on CASSANDRA-16764:
---
You're right on both accounts. Brain moving a little slowly today. Thanks!
> Compaction repeatedly fails validateReallocation exception
> --
>
> Key: CASSANDRA-16764
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16764
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Compaction/LCS
>Reporter: Richard Hesse
>Priority: Normal
>
> I have a few nodes in my ring that are stuck repeatedly trying to compact the
> same tables over and over again. I've run through the usual trick of rolling
> restarts, and it doesn't seem to help. This exception is logged on the nodes:
> {code}
> ERROR [CompactionExecutor:6] 2021-06-25 20:28:30,001 CassandraDaemon.java:244
> - Exception in thread Thread[CompactionExecutor:6,1,main]
> java.lang.RuntimeException: null
> at
> org.apache.cassandra.io.util.DataOutputBuffer.validateReallocation(DataOutputBuffer.java:134)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.calculateNewSize(DataOutputBuffer.java:152)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.expandToFit(DataOutputBuffer.java:159)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.doFlush(DataOutputBuffer.java:119)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:132)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:151)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.utils.ByteBufferUtil.writeWithVIntLength(ByteBufferUtil.java:296)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.marshal.AbstractType.writeValue(AbstractType.java:426)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ClusteringPrefix$Serializer.serializeValuesWithoutSize(ClusteringPrefix.java:323)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Clustering$Serializer.serialize(Clustering.java:131)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ClusteringPrefix$Serializer.serialize(ClusteringPrefix.java:266)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Serializers$NewFormatSerializer.serialize(Serializers.java:167)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Serializers$NewFormatSerializer.serialize(Serializers.java:154)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.IndexInfo$Serializer.serialize(IndexInfo.java:103)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.IndexInfo$Serializer.serialize(IndexInfo.java:82)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ColumnIndex.addIndexBlock(ColumnIndex.java:216)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.db.ColumnIndex.add(ColumnIndex.java:264)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ColumnIndex.buildRowIndex(ColumnIndex.java:111)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:173)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:136)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.writers.MaxSSTableSizeWriter.realAppend(MaxSSTableSizeWriter.java:98)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:143)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:204)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
>
[jira] [Commented] (CASSANDRA-16764) Compaction repeatedly fails validateReallocation exception
[
https://issues.apache.org/jira/browse/CASSANDRA-16764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17382137#comment-17382137
]
Brandon Williams commented on CASSANDRA-16764:
--
bq. I'll need to pass a ~2GB partition key value to nodetool
My understanding is the problem exists in a value, not the PK? So the issue
would be finding the PK for the value, which the heap dump may help reveal, but
also we may be able to determine the sstable directy looking at what sstables
were compacting before the error is thrown.
> Compaction repeatedly fails validateReallocation exception
> --
>
> Key: CASSANDRA-16764
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16764
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Compaction/LCS
>Reporter: Richard Hesse
>Priority: Normal
>
> I have a few nodes in my ring that are stuck repeatedly trying to compact the
> same tables over and over again. I've run through the usual trick of rolling
> restarts, and it doesn't seem to help. This exception is logged on the nodes:
> {code}
> ERROR [CompactionExecutor:6] 2021-06-25 20:28:30,001 CassandraDaemon.java:244
> - Exception in thread Thread[CompactionExecutor:6,1,main]
> java.lang.RuntimeException: null
> at
> org.apache.cassandra.io.util.DataOutputBuffer.validateReallocation(DataOutputBuffer.java:134)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.calculateNewSize(DataOutputBuffer.java:152)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.expandToFit(DataOutputBuffer.java:159)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.doFlush(DataOutputBuffer.java:119)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:132)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:151)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.utils.ByteBufferUtil.writeWithVIntLength(ByteBufferUtil.java:296)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.marshal.AbstractType.writeValue(AbstractType.java:426)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ClusteringPrefix$Serializer.serializeValuesWithoutSize(ClusteringPrefix.java:323)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Clustering$Serializer.serialize(Clustering.java:131)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ClusteringPrefix$Serializer.serialize(ClusteringPrefix.java:266)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Serializers$NewFormatSerializer.serialize(Serializers.java:167)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Serializers$NewFormatSerializer.serialize(Serializers.java:154)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.IndexInfo$Serializer.serialize(IndexInfo.java:103)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.IndexInfo$Serializer.serialize(IndexInfo.java:82)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ColumnIndex.addIndexBlock(ColumnIndex.java:216)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.db.ColumnIndex.add(ColumnIndex.java:264)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ColumnIndex.buildRowIndex(ColumnIndex.java:111)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:173)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:136)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.writers.MaxSSTableSizeWriter.realAppend(MaxSSTableSizeWriter.java:98)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:143)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:204)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
>
[jira] [Commented] (CASSANDRA-16764) Compaction repeatedly fails validateReallocation exception
[
https://issues.apache.org/jira/browse/CASSANDRA-16764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17382133#comment-17382133
]
Richard Hesse commented on CASSANDRA-16764:
---
[~brandon.williams] I like the sstablesplit idea, that's probably the best
approach.
There's a snag I'll hit before trying. I'll need to pass a ~2GB partition key
value to nodetool, and the Linux kernel has a hardcoded maximum single argument
length limit (MAX_ARG_STRLEN) of 32 pages. Is there an equivalent cqlsh command
to find the sstable for a record? Otherwise I'm guessing which sstable the
value is in. I could narrow it down via tools like strace, but knowing the
exact sstable would definitely help. Speaking of help, thanks for all of yours!
> Compaction repeatedly fails validateReallocation exception
> --
>
> Key: CASSANDRA-16764
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16764
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Compaction/LCS
>Reporter: Richard Hesse
>Priority: Normal
>
> I have a few nodes in my ring that are stuck repeatedly trying to compact the
> same tables over and over again. I've run through the usual trick of rolling
> restarts, and it doesn't seem to help. This exception is logged on the nodes:
> {code}
> ERROR [CompactionExecutor:6] 2021-06-25 20:28:30,001 CassandraDaemon.java:244
> - Exception in thread Thread[CompactionExecutor:6,1,main]
> java.lang.RuntimeException: null
> at
> org.apache.cassandra.io.util.DataOutputBuffer.validateReallocation(DataOutputBuffer.java:134)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.calculateNewSize(DataOutputBuffer.java:152)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.expandToFit(DataOutputBuffer.java:159)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.doFlush(DataOutputBuffer.java:119)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:132)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:151)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.utils.ByteBufferUtil.writeWithVIntLength(ByteBufferUtil.java:296)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.marshal.AbstractType.writeValue(AbstractType.java:426)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ClusteringPrefix$Serializer.serializeValuesWithoutSize(ClusteringPrefix.java:323)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Clustering$Serializer.serialize(Clustering.java:131)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ClusteringPrefix$Serializer.serialize(ClusteringPrefix.java:266)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Serializers$NewFormatSerializer.serialize(Serializers.java:167)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Serializers$NewFormatSerializer.serialize(Serializers.java:154)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.IndexInfo$Serializer.serialize(IndexInfo.java:103)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.IndexInfo$Serializer.serialize(IndexInfo.java:82)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ColumnIndex.addIndexBlock(ColumnIndex.java:216)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.db.ColumnIndex.add(ColumnIndex.java:264)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ColumnIndex.buildRowIndex(ColumnIndex.java:111)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:173)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:136)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.writers.MaxSSTableSizeWriter.realAppend(MaxSSTableSizeWriter.java:98)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:143)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:204)
>
[jira] [Commented] (CASSANDRA-16764) Compaction repeatedly fails validateReallocation exception
[
https://issues.apache.org/jira/browse/CASSANDRA-16764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17382083#comment-17382083
]
Brandon Williams commented on CASSANDRA-16764:
--
One thing you may be able to do is run sstablesplit offline on the offending
sstable with an output size less than 2G. Since it won't split partitions,
this should mean one of the resulting sstables is ~= 2GB with your problem
partition, which you can then throw away and recompact the others.
> Compaction repeatedly fails validateReallocation exception
> --
>
> Key: CASSANDRA-16764
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16764
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Compaction/LCS
>Reporter: Richard Hesse
>Priority: Normal
>
> I have a few nodes in my ring that are stuck repeatedly trying to compact the
> same tables over and over again. I've run through the usual trick of rolling
> restarts, and it doesn't seem to help. This exception is logged on the nodes:
> {code}
> ERROR [CompactionExecutor:6] 2021-06-25 20:28:30,001 CassandraDaemon.java:244
> - Exception in thread Thread[CompactionExecutor:6,1,main]
> java.lang.RuntimeException: null
> at
> org.apache.cassandra.io.util.DataOutputBuffer.validateReallocation(DataOutputBuffer.java:134)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.calculateNewSize(DataOutputBuffer.java:152)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.expandToFit(DataOutputBuffer.java:159)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.doFlush(DataOutputBuffer.java:119)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:132)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:151)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.utils.ByteBufferUtil.writeWithVIntLength(ByteBufferUtil.java:296)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.marshal.AbstractType.writeValue(AbstractType.java:426)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ClusteringPrefix$Serializer.serializeValuesWithoutSize(ClusteringPrefix.java:323)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Clustering$Serializer.serialize(Clustering.java:131)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ClusteringPrefix$Serializer.serialize(ClusteringPrefix.java:266)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Serializers$NewFormatSerializer.serialize(Serializers.java:167)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Serializers$NewFormatSerializer.serialize(Serializers.java:154)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.IndexInfo$Serializer.serialize(IndexInfo.java:103)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.IndexInfo$Serializer.serialize(IndexInfo.java:82)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ColumnIndex.addIndexBlock(ColumnIndex.java:216)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.db.ColumnIndex.add(ColumnIndex.java:264)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ColumnIndex.buildRowIndex(ColumnIndex.java:111)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:173)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:136)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.writers.MaxSSTableSizeWriter.realAppend(MaxSSTableSizeWriter.java:98)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:143)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:204)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85)
>
[jira] [Commented] (CASSANDRA-16764) Compaction repeatedly fails validateReallocation exception
[
https://issues.apache.org/jira/browse/CASSANDRA-16764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17382073#comment-17382073
]
Brandon Williams commented on CASSANDRA-16764:
--
Scrub detects corruption, but in this case I don't think you have any. I think
you have the problem row in more than one sstable, and when it tries to compact
the result is too large, but in any individual sstable it's fine. Can you
confirm how many sstables this is in? (nodetool getsstables may help there)
> Compaction repeatedly fails validateReallocation exception
> --
>
> Key: CASSANDRA-16764
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16764
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Compaction/LCS
>Reporter: Richard Hesse
>Priority: Normal
>
> I have a few nodes in my ring that are stuck repeatedly trying to compact the
> same tables over and over again. I've run through the usual trick of rolling
> restarts, and it doesn't seem to help. This exception is logged on the nodes:
> {code}
> ERROR [CompactionExecutor:6] 2021-06-25 20:28:30,001 CassandraDaemon.java:244
> - Exception in thread Thread[CompactionExecutor:6,1,main]
> java.lang.RuntimeException: null
> at
> org.apache.cassandra.io.util.DataOutputBuffer.validateReallocation(DataOutputBuffer.java:134)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.calculateNewSize(DataOutputBuffer.java:152)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.expandToFit(DataOutputBuffer.java:159)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.doFlush(DataOutputBuffer.java:119)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:132)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:151)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.utils.ByteBufferUtil.writeWithVIntLength(ByteBufferUtil.java:296)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.marshal.AbstractType.writeValue(AbstractType.java:426)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ClusteringPrefix$Serializer.serializeValuesWithoutSize(ClusteringPrefix.java:323)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Clustering$Serializer.serialize(Clustering.java:131)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ClusteringPrefix$Serializer.serialize(ClusteringPrefix.java:266)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Serializers$NewFormatSerializer.serialize(Serializers.java:167)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Serializers$NewFormatSerializer.serialize(Serializers.java:154)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.IndexInfo$Serializer.serialize(IndexInfo.java:103)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.IndexInfo$Serializer.serialize(IndexInfo.java:82)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ColumnIndex.addIndexBlock(ColumnIndex.java:216)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.db.ColumnIndex.add(ColumnIndex.java:264)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ColumnIndex.buildRowIndex(ColumnIndex.java:111)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:173)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:136)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.writers.MaxSSTableSizeWriter.realAppend(MaxSSTableSizeWriter.java:98)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:143)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:204)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
>
[jira] [Commented] (CASSANDRA-16764) Compaction repeatedly fails validateReallocation exception
[
https://issues.apache.org/jira/browse/CASSANDRA-16764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17381644#comment-17381644
]
Richard Hesse commented on CASSANDRA-16764:
---
[~brandon.williams] Or maybe this is something that nodetool scrub could be
patched to detect and fix? (eject the non-compacting record)
> Compaction repeatedly fails validateReallocation exception
> --
>
> Key: CASSANDRA-16764
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16764
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Compaction/LCS
>Reporter: Richard Hesse
>Priority: Normal
>
> I have a few nodes in my ring that are stuck repeatedly trying to compact the
> same tables over and over again. I've run through the usual trick of rolling
> restarts, and it doesn't seem to help. This exception is logged on the nodes:
> {code}
> ERROR [CompactionExecutor:6] 2021-06-25 20:28:30,001 CassandraDaemon.java:244
> - Exception in thread Thread[CompactionExecutor:6,1,main]
> java.lang.RuntimeException: null
> at
> org.apache.cassandra.io.util.DataOutputBuffer.validateReallocation(DataOutputBuffer.java:134)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.calculateNewSize(DataOutputBuffer.java:152)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.expandToFit(DataOutputBuffer.java:159)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.doFlush(DataOutputBuffer.java:119)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:132)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:151)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.utils.ByteBufferUtil.writeWithVIntLength(ByteBufferUtil.java:296)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.marshal.AbstractType.writeValue(AbstractType.java:426)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ClusteringPrefix$Serializer.serializeValuesWithoutSize(ClusteringPrefix.java:323)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Clustering$Serializer.serialize(Clustering.java:131)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ClusteringPrefix$Serializer.serialize(ClusteringPrefix.java:266)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Serializers$NewFormatSerializer.serialize(Serializers.java:167)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Serializers$NewFormatSerializer.serialize(Serializers.java:154)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.IndexInfo$Serializer.serialize(IndexInfo.java:103)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.IndexInfo$Serializer.serialize(IndexInfo.java:82)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ColumnIndex.addIndexBlock(ColumnIndex.java:216)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.db.ColumnIndex.add(ColumnIndex.java:264)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ColumnIndex.buildRowIndex(ColumnIndex.java:111)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:173)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:136)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.writers.MaxSSTableSizeWriter.realAppend(MaxSSTableSizeWriter.java:98)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:143)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:204)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
>
[jira] [Commented] (CASSANDRA-16764) Compaction repeatedly fails validateReallocation exception
[
https://issues.apache.org/jira/browse/CASSANDRA-16764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17376067#comment-17376067
]
Richard Hesse commented on CASSANDRA-16764:
---
Right, I was afraid of that. Given that sstables are immutable, we can't
exactly go in and pop-out that bad record. Or is there a way to pipe the output
of sstabledump into another tool, stripping out the bad object with sstabledump?
> Compaction repeatedly fails validateReallocation exception
> --
>
> Key: CASSANDRA-16764
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16764
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Compaction/LCS
>Reporter: Richard Hesse
>Priority: Normal
>
> I have a few nodes in my ring that are stuck repeatedly trying to compact the
> same tables over and over again. I've run through the usual trick of rolling
> restarts, and it doesn't seem to help. This exception is logged on the nodes:
> {code}
> ERROR [CompactionExecutor:6] 2021-06-25 20:28:30,001 CassandraDaemon.java:244
> - Exception in thread Thread[CompactionExecutor:6,1,main]
> java.lang.RuntimeException: null
> at
> org.apache.cassandra.io.util.DataOutputBuffer.validateReallocation(DataOutputBuffer.java:134)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.calculateNewSize(DataOutputBuffer.java:152)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.expandToFit(DataOutputBuffer.java:159)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.doFlush(DataOutputBuffer.java:119)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:132)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:151)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.utils.ByteBufferUtil.writeWithVIntLength(ByteBufferUtil.java:296)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.marshal.AbstractType.writeValue(AbstractType.java:426)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ClusteringPrefix$Serializer.serializeValuesWithoutSize(ClusteringPrefix.java:323)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Clustering$Serializer.serialize(Clustering.java:131)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ClusteringPrefix$Serializer.serialize(ClusteringPrefix.java:266)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Serializers$NewFormatSerializer.serialize(Serializers.java:167)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Serializers$NewFormatSerializer.serialize(Serializers.java:154)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.IndexInfo$Serializer.serialize(IndexInfo.java:103)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.IndexInfo$Serializer.serialize(IndexInfo.java:82)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ColumnIndex.addIndexBlock(ColumnIndex.java:216)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.db.ColumnIndex.add(ColumnIndex.java:264)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ColumnIndex.buildRowIndex(ColumnIndex.java:111)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:173)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:136)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.writers.MaxSSTableSizeWriter.realAppend(MaxSSTableSizeWriter.java:98)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:143)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:204)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
>
[jira] [Commented] (CASSANDRA-16764) Compaction repeatedly fails validateReallocation exception
[
https://issues.apache.org/jira/browse/CASSANDRA-16764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17371325#comment-17371325
]
Brandon Williams commented on CASSANDRA-16764:
--
Unfortunately for a tombstone to remove data, it has to compact with it, so I
think some manual process will need to be worked out here.
> Compaction repeatedly fails validateReallocation exception
> --
>
> Key: CASSANDRA-16764
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16764
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Compaction/LCS
>Reporter: Richard Hesse
>Priority: Normal
>
> I have a few nodes in my ring that are stuck repeatedly trying to compact the
> same tables over and over again. I've run through the usual trick of rolling
> restarts, and it doesn't seem to help. This exception is logged on the nodes:
> {code}
> ERROR [CompactionExecutor:6] 2021-06-25 20:28:30,001 CassandraDaemon.java:244
> - Exception in thread Thread[CompactionExecutor:6,1,main]
> java.lang.RuntimeException: null
> at
> org.apache.cassandra.io.util.DataOutputBuffer.validateReallocation(DataOutputBuffer.java:134)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.calculateNewSize(DataOutputBuffer.java:152)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.expandToFit(DataOutputBuffer.java:159)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.doFlush(DataOutputBuffer.java:119)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:132)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:151)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.utils.ByteBufferUtil.writeWithVIntLength(ByteBufferUtil.java:296)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.marshal.AbstractType.writeValue(AbstractType.java:426)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ClusteringPrefix$Serializer.serializeValuesWithoutSize(ClusteringPrefix.java:323)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Clustering$Serializer.serialize(Clustering.java:131)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ClusteringPrefix$Serializer.serialize(ClusteringPrefix.java:266)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Serializers$NewFormatSerializer.serialize(Serializers.java:167)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Serializers$NewFormatSerializer.serialize(Serializers.java:154)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.IndexInfo$Serializer.serialize(IndexInfo.java:103)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.IndexInfo$Serializer.serialize(IndexInfo.java:82)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ColumnIndex.addIndexBlock(ColumnIndex.java:216)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.db.ColumnIndex.add(ColumnIndex.java:264)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ColumnIndex.buildRowIndex(ColumnIndex.java:111)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:173)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:136)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.writers.MaxSSTableSizeWriter.realAppend(MaxSSTableSizeWriter.java:98)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:143)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:204)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
>
[jira] [Commented] (CASSANDRA-16764) Compaction repeatedly fails validateReallocation exception
[
https://issues.apache.org/jira/browse/CASSANDRA-16764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17370973#comment-17370973
]
Richard Hesse commented on CASSANDRA-16764:
---
[~dcapwell] we've deleted the bad object, but the error is still being thrown
on compaction. I guess it will take a while for the tombstone process to make
its way through? Or is this something that will require some manual commands to
recover from? Thanks again for your help and quick response!
> Compaction repeatedly fails validateReallocation exception
> --
>
> Key: CASSANDRA-16764
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16764
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Compaction/LCS
>Reporter: Richard Hesse
>Priority: Normal
>
> I have a few nodes in my ring that are stuck repeatedly trying to compact the
> same tables over and over again. I've run through the usual trick of rolling
> restarts, and it doesn't seem to help. This exception is logged on the nodes:
> {code}
> ERROR [CompactionExecutor:6] 2021-06-25 20:28:30,001 CassandraDaemon.java:244
> - Exception in thread Thread[CompactionExecutor:6,1,main]
> java.lang.RuntimeException: null
> at
> org.apache.cassandra.io.util.DataOutputBuffer.validateReallocation(DataOutputBuffer.java:134)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.calculateNewSize(DataOutputBuffer.java:152)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.expandToFit(DataOutputBuffer.java:159)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.doFlush(DataOutputBuffer.java:119)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:132)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:151)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.utils.ByteBufferUtil.writeWithVIntLength(ByteBufferUtil.java:296)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.marshal.AbstractType.writeValue(AbstractType.java:426)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ClusteringPrefix$Serializer.serializeValuesWithoutSize(ClusteringPrefix.java:323)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Clustering$Serializer.serialize(Clustering.java:131)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ClusteringPrefix$Serializer.serialize(ClusteringPrefix.java:266)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Serializers$NewFormatSerializer.serialize(Serializers.java:167)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Serializers$NewFormatSerializer.serialize(Serializers.java:154)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.IndexInfo$Serializer.serialize(IndexInfo.java:103)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.IndexInfo$Serializer.serialize(IndexInfo.java:82)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ColumnIndex.addIndexBlock(ColumnIndex.java:216)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.db.ColumnIndex.add(ColumnIndex.java:264)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ColumnIndex.buildRowIndex(ColumnIndex.java:111)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:173)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:136)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.writers.MaxSSTableSizeWriter.realAppend(MaxSSTableSizeWriter.java:98)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:143)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:204)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85)
>
[jira] [Commented] (CASSANDRA-16764) Compaction repeatedly fails validateReallocation exception
[
https://issues.apache.org/jira/browse/CASSANDRA-16764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17370928#comment-17370928
]
Richard Hesse commented on CASSANDRA-16764:
---
Looking at the heap dump, there's definitely some application level garbage
that made its way in there. Yep, it's a 2GB column value.
> Compaction repeatedly fails validateReallocation exception
> --
>
> Key: CASSANDRA-16764
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16764
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Compaction/LCS
>Reporter: Richard Hesse
>Priority: Normal
>
> I have a few nodes in my ring that are stuck repeatedly trying to compact the
> same tables over and over again. I've run through the usual trick of rolling
> restarts, and it doesn't seem to help. This exception is logged on the nodes:
> {code}
> ERROR [CompactionExecutor:6] 2021-06-25 20:28:30,001 CassandraDaemon.java:244
> - Exception in thread Thread[CompactionExecutor:6,1,main]
> java.lang.RuntimeException: null
> at
> org.apache.cassandra.io.util.DataOutputBuffer.validateReallocation(DataOutputBuffer.java:134)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.calculateNewSize(DataOutputBuffer.java:152)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.expandToFit(DataOutputBuffer.java:159)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.doFlush(DataOutputBuffer.java:119)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:132)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:151)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.utils.ByteBufferUtil.writeWithVIntLength(ByteBufferUtil.java:296)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.marshal.AbstractType.writeValue(AbstractType.java:426)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ClusteringPrefix$Serializer.serializeValuesWithoutSize(ClusteringPrefix.java:323)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Clustering$Serializer.serialize(Clustering.java:131)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ClusteringPrefix$Serializer.serialize(ClusteringPrefix.java:266)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Serializers$NewFormatSerializer.serialize(Serializers.java:167)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Serializers$NewFormatSerializer.serialize(Serializers.java:154)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.IndexInfo$Serializer.serialize(IndexInfo.java:103)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.IndexInfo$Serializer.serialize(IndexInfo.java:82)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ColumnIndex.addIndexBlock(ColumnIndex.java:216)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.db.ColumnIndex.add(ColumnIndex.java:264)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ColumnIndex.buildRowIndex(ColumnIndex.java:111)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:173)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:136)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.writers.MaxSSTableSizeWriter.realAppend(MaxSSTableSizeWriter.java:98)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:143)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:204)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
>
[jira] [Commented] (CASSANDRA-16764) Compaction repeatedly fails validateReallocation exception
[
https://issues.apache.org/jira/browse/CASSANDRA-16764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17370845#comment-17370845
]
Richard Hesse commented on CASSANDRA-16764:
---
[~dcapwell] Thanks for the quick response! The clustering key is ASCII text of
a URI path. I guess we had some customers pick some long ones. I'll try to grab
a heap dump later tonight.
> Compaction repeatedly fails validateReallocation exception
> --
>
> Key: CASSANDRA-16764
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16764
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Compaction/LCS
>Reporter: Richard Hesse
>Priority: Normal
>
> I have a few nodes in my ring that are stuck repeatedly trying to compact the
> same tables over and over again. I've run through the usual trick of rolling
> restarts, and it doesn't seem to help. This exception is logged on the nodes:
> {code}
> ERROR [CompactionExecutor:6] 2021-06-25 20:28:30,001 CassandraDaemon.java:244
> - Exception in thread Thread[CompactionExecutor:6,1,main]
> java.lang.RuntimeException: null
> at
> org.apache.cassandra.io.util.DataOutputBuffer.validateReallocation(DataOutputBuffer.java:134)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.calculateNewSize(DataOutputBuffer.java:152)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.expandToFit(DataOutputBuffer.java:159)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.doFlush(DataOutputBuffer.java:119)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:132)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:151)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.utils.ByteBufferUtil.writeWithVIntLength(ByteBufferUtil.java:296)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.marshal.AbstractType.writeValue(AbstractType.java:426)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ClusteringPrefix$Serializer.serializeValuesWithoutSize(ClusteringPrefix.java:323)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Clustering$Serializer.serialize(Clustering.java:131)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ClusteringPrefix$Serializer.serialize(ClusteringPrefix.java:266)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Serializers$NewFormatSerializer.serialize(Serializers.java:167)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Serializers$NewFormatSerializer.serialize(Serializers.java:154)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.IndexInfo$Serializer.serialize(IndexInfo.java:103)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.IndexInfo$Serializer.serialize(IndexInfo.java:82)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ColumnIndex.addIndexBlock(ColumnIndex.java:216)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.db.ColumnIndex.add(ColumnIndex.java:264)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ColumnIndex.buildRowIndex(ColumnIndex.java:111)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:173)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:136)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.writers.MaxSSTableSizeWriter.realAppend(MaxSSTableSizeWriter.java:98)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:143)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:204)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
>
[jira] [Commented] (CASSANDRA-16764) Compaction repeatedly fails validateReallocation exception
[
https://issues.apache.org/jira/browse/CASSANDRA-16764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17370776#comment-17370776
]
David Capwell commented on CASSANDRA-16764:
---
this is what I see in the code
{code}
static final int MAX_ARRAY_SIZE = Integer.MAX_VALUE - 8;
...
long validateReallocation(long newSize)
{
int saturatedSize = saturatedArraySizeCast(newSize);
if (saturatedSize <= capacity())
throw new RuntimeException();
return saturatedSize;
}
{code}
this is called by
{code}
long calculateNewSize(long count)
{
long capacity = capacity();
//Both sides of this max expression need to use long arithmetic to
avoid integer overflow
//count and capacity are longs so that ensures it right now.
long newSize = capacity + count;
//For large buffers don't double, increase by 50%
if (capacity > 1024L * 1024L * DOUBLING_THRESHOLD)
newSize = Math.max((capacity * 3L) / 2L, newSize);
else
newSize = Math.max(capacity * 2L, newSize);
return validateReallocation(newSize);
}
{code}
which is called by
{code}
protected void expandToFit(long count)
{
if (count <= 0)
return;
ByteBuffer newBuffer =
ByteBuffer.allocate(checkedArraySizeCast(calculateNewSize(count)));
buffer.flip();
newBuffer.put(buffer);
buffer = newBuffer;
}
{code}
This implies to me that the buffer is MAX_ARRAY_SIZE, at this point we are no
longer able to expand (should return a better error in this case). I say this
as calculateNewSize should not be able to return a value < capacity, so
saturatedArraySizeCast trimming to MAX_ARRAY_SIZE is what should trigger this.
If this is the case then it means the clustering key has a column larger than
{code:java}
Integer.MAX_VALUE - 8
{code}
bytes; what type is used for this table?
[~richardchesse] if you could take a look at a heap dump while this is
happening it would help see what's going on. If you have data ~
Integer.MAX_VALUE then I would expect other issues going on (such as GC issues).
> Compaction repeatedly fails validateReallocation exception
> --
>
> Key: CASSANDRA-16764
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16764
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Compaction/LCS
>Reporter: Richard Hesse
>Priority: Normal
>
> I have a few nodes in my ring that are stuck repeatedly trying to compact the
> same tables over and over again. I've run through the usual trick of rolling
> restarts, and it doesn't seem to help. This exception is logged on the nodes:
> {code}
> ERROR [CompactionExecutor:6] 2021-06-25 20:28:30,001 CassandraDaemon.java:244
> - Exception in thread Thread[CompactionExecutor:6,1,main]
> java.lang.RuntimeException: null
> at
> org.apache.cassandra.io.util.DataOutputBuffer.validateReallocation(DataOutputBuffer.java:134)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.calculateNewSize(DataOutputBuffer.java:152)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.expandToFit(DataOutputBuffer.java:159)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.doFlush(DataOutputBuffer.java:119)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:132)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:151)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.utils.ByteBufferUtil.writeWithVIntLength(ByteBufferUtil.java:296)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.marshal.AbstractType.writeValue(AbstractType.java:426)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ClusteringPrefix$Serializer.serializeValuesWithoutSize(ClusteringPrefix.java:323)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Clustering$Serializer.serialize(Clustering.java:131)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.ClusteringPrefix$Serializer.serialize(ClusteringPrefix.java:266)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Serializers$NewFormatSerializer.serialize(Serializers.java:167)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
> at
> org.apache.cassandra.db.Serializers$NewFormatSerializer.serialize(Serializers.java:154)
> ~[apache-cassandra-3.11.10.jar:3.11.10]
>
