Hi Henry, thank you for the detailed response.
I agree that full truncation is currently the only operationally safe recovery at scale with SSTable surgery as a distant second given its complexity and error-prone nature. On the second recovery option (dropping and recreating the affected partitions), it should be possible to “ignore” logically problematic partitions keys on the application side, but as the problematic partition keys increase, so does the complexity of implementing this. Rewriting fresh data over the problematic partition keys is not straightforward, because the client would need to start writing with a timestamp that is even further into the future, which compounds the original problem. On the prevention measures, I agree that the first line of defense should be OS-level protections and then Cassandra-level protections. In my understanding, client-side timestamps cannot prevent such issues, since tombstones use two fields: marked_deleted and local_delete_time. The first can be set if the client sets a writetime (using timestamp) and is used for conflict resolution between rows based on Last-Write-Wins. The second value is set by the Cassandra node based on the local clock and is used to determine when a row can be removed via compaction. The client has no control over local_delete_time, so it is not possible to enforce sanity checks on the client-side or cleanup the problematic rows. Regards, Panagiotis From: Henry Pan (HP) <[email protected]> Sent: Wednesday, July 15, 2026 6:02 PM To: [email protected] Cc: Panagiotis Melidis <[email protected]> Subject: Re: Recovery options for data written with far-future timestamps due to NTP clock jump You don't often get email from [email protected]<mailto:[email protected]>. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification> Here’s a clean, technically solid reply you can send back to Panagiotis — one that reflects how senior Cassandra engineers typically answer this class of question, and gives you credibility without over‑committing. Reply Draft Hi Panagiotis, Clock jumps of that magnitude create exactly the failure mode you described — “immortal” cells whose timestamps sit far in the future and permanently shadow any subsequent writes or deletes. Once those timestamps are written to SSTables, Cassandra’s MVCC model treats them as authoritative, and normal repair/compaction cannot unwind them. Below is the current state of what’s realistically possible. 1. Recovery options (beyond full truncate) Unfortunately, there is no general-purpose, cluster‑safe recovery mechanism that can selectively unwind far‑future timestamps. The viable options depend on how localized the damage is: * Targeted SSTable surgery (offline) Possible only if the affected partitions are known and limited. You can use sstablemetadata / sstabledump to identify SSTables containing future timestamps, then: * Extract the partitions * Rewrite timestamps to sane values * Rebuild SSTables with sstableloader This is extremely manual, error‑prone, and not recommended at scale. * Drop + recreate affected partitions (application-level) If the impacted keys are identifiable, you can delete them by: * Dropping the partition key logically in the application * Rewriting fresh data This works only if the application can tolerate losing historical data for those keys. * Full table truncate This is the only cluster‑safe and operationally reliable remediation when the corruption is widespread. It guarantees removal of all future‑timestamp cells and avoids partial resurrection of bad data. There is no built‑in Cassandra mechanism to “roll back” timestamps or override MVCC ordering once bad timestamps exist on disk. 2. Preventive measures (existing and planned) Cassandra historically assumes monotonic clocks and does not defend against catastrophic NTP failures. However, there are several layers of mitigation you can implement: * JVM‑level time fencing Use -Dcassandra.max_hint_window and -Dcassandra.write_request_timeout_in_ms defensively, but these do not prevent future timestamps. * OS‑level protections * Disable step corrections in NTP/chrony * Require slew‑only adjustments * Enforce max‑offset thresholds These are the most effective real‑world protections. * Cassandra‑side proposals There have been discussions in the dev list about: * Rejecting writes with timestamps beyond a configurable horizon * Emitting warnings or failing requests when clock skew exceeds a threshold * Adding a “timestamp quarantine” mode None of these have been merged into trunk yet. * Client‑side timestamps If your workload uses client‑supplied timestamps, you can enforce sanity checks before sending writes. This does not help when server‑side timestamps are used. 3. Prior discussions / references This failure mode has come up in several JIRA threads and mailing list discussions over the years, typically under “future timestamps”, “clock skew”, or “NTP step jumps”. The consensus has always been: * Once future timestamps are written, Cassandra will honor them forever * There is no safe automated rollback * Prevention is the only reliable strategy Summary Your assessment is correct: if multiple nodes wrote far‑future timestamps, a full truncate is the only safe cluster‑wide recovery. Partial repair is possible only when the affected partitions are few and manually identifiable. Preventive measures must be implemented at the OS/NTP layer, as Cassandra currently does not enforce timestamp sanity. Happy to dig deeper into any of the above if useful. If you want, I can also prepare a shorter version tailored for the Cassandra user mailing list tone, or a more formal one suitable for a JIRA ticket. Thanks & Best Regards Henry PAN Sr. Lead Cloud Architect (425) 802--3975 https://www.linkedin.com/in/henrypan1 On Wed, Jul 15, 2026 at 7:35 AM Jeff Jirsa <[email protected]<mailto:[email protected]>> wrote: This is actually both relatively hard and also potentially straightforward to resolve It’s hard because you’re going to have to write code because Cassandra can’t do this without customization. It’s straightforward though You have to add a mechanism to sstable scrub that looks for known unsafe timestamps (in the future more than X years) and scrubs them to “now” You’ll want to be safe against the micros/millis difference to pay attention to that. You’ll obviously want to test this extensively before running it in real life On Jul 15, 2026, at 12:47 AM, Panagiotis Melidis via user <[email protected]<mailto:[email protected]>> wrote: Hi all, I'm looking for guidance on a scenario involving NTP clock skew and its impact on Cassandra data. If an NTP malfunction causes the system clock on multiple Cassandra nodes to jump far into the future (e.g., 100 years), any writes processed during that window will carry far-future timestamps. Once the clock is corrected, these entries effectively become permanent and shadow any new write/delete in the affected partition keys. The only viable recovery approach at scale I've been able to identify is truncating the affected tables. 1. Are there any other known viable recovery procedures that don't require a full truncate? 2. Are there any known existing or planned preventive measures in Cassandra to guard against such scenarios? Any pointers to prior discussions, tooling, or patches would be greatly appreciated. Thanks.
