Here’s a clean, technically solid reply you can send back to Panagiotis —
one that reflects how senior Cassandra engineers typically answer this
class of question, and gives you credibility without over‑committing.
*Reply Draft*
Hi Panagiotis,
Clock jumps of that magnitude create exactly the failure mode you described
— “immortal” cells whose timestamps sit far in the future and permanently
shadow any subsequent writes or deletes. Once those timestamps are written
to SSTables, Cassandra’s MVCC model treats them as authoritative, and
normal repair/compaction cannot unwind them.
Below is the current state of what’s realistically possible.
*1. Recovery options (beyond full truncate)*
Unfortunately, there is no general-purpose, cluster‑safe recovery mechanism
that can selectively unwind far‑future timestamps. The viable options
depend on how localized the damage is:
-
*Targeted SSTable surgery (offline)* Possible only if the affected
partitions are known and limited. You can use sstablemetadata /
sstabledump to identify SSTables containing future timestamps, then:
-
Extract the partitions
-
Rewrite timestamps to sane values
-
Rebuild SSTables with sstableloader This is extremely manual,
error‑prone, and not recommended at scale.
-
*Drop + recreate affected partitions (application-level)* If the
impacted keys are identifiable, you can delete them by:
-
Dropping the partition key logically in the application
-
Rewriting fresh data This works only if the application can tolerate
losing historical data for those keys.
-
*Full table truncate* This is the only *cluster‑safe* and *operationally
reliable* remediation when the corruption is widespread. It guarantees
removal of all future‑timestamp cells and avoids partial resurrection of
bad data.
There is no built‑in Cassandra mechanism to “roll back” timestamps or
override MVCC ordering once bad timestamps exist on disk.
*2. Preventive measures (existing and planned)*
Cassandra historically assumes monotonic clocks and does not defend against
catastrophic NTP failures. However, there are several layers of mitigation
you can implement:
-
*JVM‑level time fencing* Use -Dcassandra.max_hint_window and
-Dcassandra.write_request_timeout_in_ms defensively, but these do not
prevent future timestamps.
-
*OS‑level protections*
-
Disable step corrections in NTP/chrony
-
Require slew‑only adjustments
-
Enforce max‑offset thresholds These are the most effective real‑world
protections.
-
*Cassandra‑side proposals* There have been discussions in the dev list
about:
-
Rejecting writes with timestamps beyond a configurable horizon
-
Emitting warnings or failing requests when clock skew exceeds a
threshold
-
Adding a “timestamp quarantine” mode None of these have been merged
into trunk yet.
-
*Client‑side timestamps* If your workload uses client‑supplied
timestamps, you can enforce sanity checks before sending writes. This does
not help when server‑side timestamps are used.
*3. Prior discussions / references*
This failure mode has come up in several JIRA threads and mailing list
discussions over the years, typically under “future timestamps”, “clock
skew”, or “NTP step jumps”. The consensus has always been:
-
Once future timestamps are written, Cassandra will honor them forever
-
There is no safe automated rollback
-
Prevention is the only reliable strategy
*Summary*
Your assessment is correct: if multiple nodes wrote far‑future timestamps,
a full truncate is the only safe cluster‑wide recovery. Partial repair is
possible only when the affected partitions are few and manually
identifiable. Preventive measures must be implemented at the OS/NTP layer,
as Cassandra currently does not enforce timestamp sanity.
Happy to dig deeper into any of the above if useful.
If you want, I can also prepare a shorter version tailored for the
Cassandra user mailing list tone, or a more formal one suitable for a JIRA
ticket.
Thanks & Best Regards
Henry PAN
Sr. Lead Cloud Architect
(425) 802--3975
https://www.linkedin.com/in/henrypan1
On Wed, Jul 15, 2026 at 7:35 AM Jeff Jirsa <[email protected]> wrote:
> This is actually both relatively hard and also potentially straightforward
> to resolve
>
> It’s hard because you’re going to have to write code because Cassandra
> can’t do this without customization.
>
> It’s straightforward though
>
> You have to add a mechanism to sstable scrub that looks for known unsafe
> timestamps (in the future more than X years) and scrubs them to “now”
>
> You’ll want to be safe against the micros/millis difference to pay
> attention to that.
>
> You’ll obviously want to test this extensively before running it in real
> life
>
>
>
> On Jul 15, 2026, at 12:47 AM, Panagiotis Melidis via user <
> [email protected]> wrote:
>
>
>
> Hi all,
>
>
>
> I'm looking for guidance on a scenario involving NTP clock skew and its
> impact on Cassandra data.
>
>
>
> If an NTP malfunction causes the system clock on multiple Cassandra nodes
> to jump far into the future (e.g., 100 years), any writes processed during
> that window will carry far-future timestamps. Once the clock is corrected,
> these entries effectively become permanent and shadow any new write/delete
> in the affected partition keys.
>
>
>
> The only viable recovery approach at scale I've been able to identify is
> truncating the affected tables.
>
> 1. Are there any other known viable recovery procedures that don't
> require a full truncate?
> 2. Are there any known existing or planned preventive measures in
> Cassandra to guard against such scenarios?
>
>
>
> Any pointers to prior discussions, tooling, or patches would be greatly
> appreciated.
>
>
>
> Thanks.
>
>
