[
https://issues.apache.org/jira/browse/CASSANDRA-21729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18124208#comment-18124208
]
Andy Salnikov commented on CASSANDRA-21729:
-------------------------------------------
[~paulo], what could be an explanation for having negative timestamps if one
never does DELETE .. USING with negative timestamps (this is what I think
original report stated)?
Could anything else lead to the same situation, e.g. reader thinks that it sees
DeletionTime record, but it is some other type of record (I know nothing about
how things are encoded, just trying to imagine all possible scenarios)?
99.22% success rate with the proposed fix is not super impressive. Would it be
possible to improve by trying to read ahead and maybe figuring out whether the
next bytes belong to DeletionTime or something else?
> Negative timestamps present in historical clusters create unsolvable
> corruptions
> --------------------------------------------------------------------------------
>
> Key: CASSANDRA-21729
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21729
> Project: Apache Cassandra
> Issue Type: Bug
> Reporter: Jerome BAROTIN
> Priority: Normal
>
> After a successful migration from 4 to 5, some exceptions happened executing
> repair or verify :
> {code:java}
> Corrupted sstable. Invalid flags found deserializing DeletionTime:
> 10100000{code}
>
>
> Launching scrub is not working as well :
> {code:java}
> INFO [CompactionExecutor:4107] 2026-09-17T10:32:28,177 OutputHandler.java:64
> - Scrub of
> BtiTableReader:bti(path='/var/lib/cassandra/data/keyspace1/table1-49c3302007d111eb95f19517b13fd7f9/da-3h2f_1en8_438xc2sy59frg5f9i3-bti-Data.db')
> complete: 59158928 partitions in new sstable and 0 empty (tombstoned)
> partitions dropped
> WARN [CompactionExecutor:4107] 2026-09-17T10:32:27,675 OutputHandler.java:74
> - Unrecoverable error while scrubbing
> BtiTableReader:bti(path='/var/lib/cassandra/data/keyspace1/table1-49c3302007d111eb95f19517b13fd7f9/da-3h2f_1en8_438xc2sy59frg5f9i3-bti-Data.db').Scrubbing
> cannot continue. The sstable will be marked for deletion. You can attempt
> manual recovery from the pre-scrub snapshot. You can also run nodetool repair
> to transfer the data from a healthy replica, if any.
> WARN [CompactionExecutor:4107] 2026-09-17T10:32:27,675 OutputHandler.java:79
> - Error reading partition 20210525-247c9972-928d-4bd2-bf83-1b012b4a44ec
> (stacktrace follows):
> org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted:
> /var/lib/cassandra/data/keyspace1/table1-49c3302007d111eb95f19517b13fd7f9/da-3h2f_1en8_438xc2sy59frg5f9i3-bti-Data.db
> at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.create(SSTableIdentityIterator.java:77)
> at
> org.apache.cassandra.io.sstable.format.SortedTableScrubber.getIterator(SortedTableScrubber.java:321)
> at
> org.apache.cassandra.io.sstable.format.SortedTableScrubber.tryAppend(SortedTableScrubber.java:291)
> at
> org.apache.cassandra.io.sstable.format.bti.BtiTableScrubber.scrubInternal(BtiTableScrubber.java:187)
> at
> org.apache.cassandra.io.sstable.format.SortedTableScrubber.scrub(SortedTableScrubber.java:193)
> at
> org.apache.cassandra.db.compaction.CompactionManager.scrubOne(CompactionManager.java:1319)
> at
> org.apache.cassandra.db.compaction.CompactionManager$2.execute(CompactionManager.java:529)
> at
> org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:457)
> at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61)
> at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Thread.java:840)
> Caused by: java.io.IOException: Corrupted sstable. Invalid flags found
> deserializing DeletionTime: 11001100
> at
> org.apache.cassandra.db.DeletionTime$Serializer.deserialize(DeletionTime.java:228)
> at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.create(SSTableIdentityIterator.java:67)
> ... 13 common frames omitted
> INFO [CompactionExecutor:4107] 2026-09-16T20:36:05,313 OutputHandler.java:64
> - Scrubbing
> BtiTableReader:bti(path='/var/lib/cassandra/data/keyspace1/table1-49c3302007d111eb95f19517b13fd7f9/da-3h2f_1en8_438xc2sy59frg5f9i3-bti-Data.db')
> (2.277TiB){code}
> This seems linked to this change:
> https://issues.apache.org/jira/browse/CASSANDRA-18648
>
> This exception below is a bit hard for the node : With some tombstones with a
> negative timestamp somewhere in an historical sstable that doesn't not fit
> these news rules introduced in 5.0, all the sstable is lost.
> In theory, it could be restored through repair, but in reality, these
> corruptions could appear in several nodes and on very big SSTables.A solution
> (enabled with a special flag maybe) to skip the tombstones on the new sstable
> during the scrub will help Cassandra administrators arbitrate the choice of
> avoiding these corrupted tombstones and keeping other data in the new
> scrubbed SSTable or rebuilding the SSTable from the cluster ?
> Also, to avoid this in the future, handling negative timestamps during
> migration from Cassandra 4 to 5 might be a good idea.
> I have checked in Cassandra 5.0.9, and it is still possible to set some
> negative timestamps with such CQL queries: DELETE FROM table USING TIMESTAMP
> -4548 WHERE key = "value1";. I don't understand what is the interest in
> pushing a negative timestamp. It shouldn't be possible, cause it creates
> corruptions.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
