[
https://issues.apache.org/jira/browse/CASSANDRA-15799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17106611#comment-17106611
]
David Capwell commented on CASSANDRA-15799:
-------------------------------------------
Thanks. Dumping context here.
What we see is that the Compact Storage for this table has: Cell.name ==
clusterKey, Cell.value == value field. We see that Cell.value has calls to
validate the type in 3.0.20, but there isn't anything for validating the
Cell.name and clustering. Based off the data the Cell.name is expected to be a
bigint, but it has more bytes than expected; which eventually corrupts
unfiltered serialization.
I will cleanup the POC to guard against this, but the open issue is "why is the
clustering key not a bigint and how do we recover"
> CorruptSSTableException when compacting a 3.0 format sstable that was
> originally created as an outcome of 2.1 sstable upgrade
> -----------------------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-15799
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15799
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Compaction, Local/SSTable
> Reporter: Sumanth Pasupuleti
> Assignee: David Capwell
> Priority: Normal
> Fix For: 3.0.x
>
>
> Below is the exception with stack trace. This issue is reproduce-able.
> {code:java}
> DEBUG [CompactionExecutor:10] 2020-05-07 19:33:34,268 CompactionTask.java:158
> - Compacting (a3ea9fc0-9099-11ea-933f-c5e852f71338)
> [/mnt/data/cassandra/data/ks/cf/md-10802-big-Data.db:level=0, ]
> ERROR [CompactionExecutor:10] 2020-05-07 19:33:34,275
> CassandraDaemon.java:208 - Exception in thread
> Thread[CompactionExecutor:10,1,RMI Runtime]
> org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted:
> /mnt/data/cassandra/data/ks/cf/md-10802-big-Data.db
> at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:105)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
> at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:30)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
> at
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
> at
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
> at
> org.apache.cassandra.utils.MergeIterator$TrivialOneToOne.computeNext(MergeIterator.java:460)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
> at
> org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:534)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
> at
> org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:394)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
> at
> org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
> at
> org.apache.cassandra.db.ColumnIndex$Builder.build(ColumnIndex.java:111)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
> at
> org.apache.cassandra.db.ColumnIndex.writeAndBuildIndex(ColumnIndex.java:52)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:165)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:126)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
> at
> org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:57)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:109)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:195)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:89)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
> at
> org.apache.cassandra.db.compaction.CompactionManager$8.runMayThrow(CompactionManager.java:675)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_231]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_231]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[na:1.8.0_231]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [na:1.8.0_231]
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:83)
> [nf-cassandra-3.0.19.8.jar:3.0.19.8]
> at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_231]
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 119
> at
> org.apache.cassandra.db.RangeTombstone$Bound$Serializer.deserialize(RangeTombstone.java:201)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
> at
> org.apache.cassandra.db.rows.UnfilteredSerializer.deserializeOne(UnfilteredSerializer.java:407)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
> at
> org.apache.cassandra.db.rows.UnfilteredSerializer.deserialize(UnfilteredSerializer.java:373)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
> at
> org.apache.cassandra.io.sstable.SSTableSimpleIterator$CurrentFormatIterator.computeNext(SSTableSimpleIterator.java:87)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
> at
> org.apache.cassandra.io.sstable.SSTableSimpleIterator$CurrentFormatIterator.computeNext(SSTableSimpleIterator.java:65)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
> at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.doCompute(SSTableIdentityIterator.java:123)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
> at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:100)
> ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
> ... 29 common frames omitted
> {code}
> Below are the logs that indicate the origin of this sstable
> {code:java}
> DEBUG [CompactionExecutor:7] 2020-05-07 18:39:59,235 CompactionTask.java:158
> - Compacting (279b8530-9092-11ea-933f-c5e852f71338)
> [/mnt/data/cassandra/data/ks/cf/ks-cf-ka-10799-Data.db:level=0,
> /mnt/data/cassandra/data/ks/cf/ks-cf-ka-10800-Data.db:level=0,
> /mnt/data/cassandra/data/ks/cf/ks-cf-ka-10801-Data.db:level=0, ]
> DEBUG [CompactionExecutor:7] 2020-05-07 18:40:01,053 CompactionTask.java:235
> - Compacted (279b8530-9092-11ea-933f-c5e852f71338) 3 sstables to
> [/mnt/data/cassandra/data/ks/cf/md-10802-big,] to level=0. 35,435,268 bytes
> to 35,317,747 (~99% of original) in 1,817ms = 18.536945MB/s. 789 total
> partitions merged to 772. Partition merge counts were {1:786, 3:1, }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
