> We use TWCS in a few of the column families that have TTL based
> time-series data, and no explicit deletes are issued. Over the time, we
> observed the disk usage has been increasing beyond the expected levels.
> Data directory in a particular node shows SSTables that are more than
> 16days old, while the bucket size is configured at 12hours, TTL is at
> 15days and GC grace at 1hour.
> Upon using sstableexpiredblockers, we got quite a few sets of blocking and
> blocked SSTables. SSTableMetadata that is shown in the output indicates
> there is an overlap in the MinTS-MaxTS period among the blocking SSTable
> and the blocked SSTables, which is preventing the older SSTables from
> getting dropped/deleted.
> Following are the possible root causes we considered
> 1. Hints - old data hints getting replayed from the coordinator node.
> We ruled this out since hints live for no more than 1 day based on our
> 2. External compactions - no external compactions were run, that could
> cause compaction of SSTables across the TWCS buckets.
> 3. Read repairs - this is ruled out as well, since we never ran
> external repairs, and read repair chance on the TWCS column families has
> been set to 0.
> 4. Application team writing data with older timestamp (in newer
> 1. We wanted to identify the specific row keys with older timestamps
> in the blocking SSTable, that could be causing this issue to occur. We
> considered using SSTable2Keys/json, however, since both the tools
> outputting the entire content/keys of the SSTable in the order of the
> they were not helpful in this case.
> 2. Since we wanted to get data on a few oldest cells with
> timestamps, we created a tool mostly based off of sstable2json, called
> sstableslicer, to output 'n' top/bottom cells in an SSTable, ordered
> on writetime/localDeletionTime. This helped us identify the specific
> in new SSTables with older timestamps, which further helped in
> debugging on
> the application end. From application team perspective, however, writing
> data with old timestamp is not a possible scenario.
> 3. Below is a sample output of sstableslicer
[image: Inline image 2]
> Looking for suggestions, especially around following two things:
> 1. Did we miss any other case in TWCS that could be causing such
> 2. Does sstableslicer seem valuable, to be included in Apache C*? If
> yes, I shall create a JIRA and submit a PR/patch for review.
> C* version we use is 2.1.17.