> Hi,
> We use TWCS in a few of the column families that have TTL based
> time-series data, and no explicit deletes are issued. Over the time, we
> observed the disk usage has been increasing beyond the expected levels.
> Data directory in a particular node shows SSTables that are more than
> 16days old, while the bucket size is configured at 12hours, TTL is at
> 15days and GC grace at 1hour.
> Upon using sstableexpiredblockers, we got quite a few sets of blocking and
> blocked SSTables. SSTableMetadata that is shown in the output indicates
> there is an overlap in the MinTS-MaxTS period among the blocking SSTable
> and the blocked SSTables, which is preventing the older SSTables from
> getting dropped/deleted.
> Following are the possible root causes we considered
>    1. Hints - old data hints getting replayed from the coordinator node.
>    We ruled this out since hints live for no more than 1 day based on our
>    configuration.
>    2. External compactions - no external compactions were run, that could
>    cause compaction of SSTables across the TWCS buckets.
>    3.  Read repairs - this is ruled out as well, since we never ran
>    external repairs, and read repair chance on the TWCS column families has
>    been set to 0.
>    4.  Application team writing data with older timestamp (in newer
>    SSTables).
>    1. We wanted to identify the specific row keys with older timestamps
>       in the blocking SSTable, that could be causing this issue to occur. We
>       considered using SSTable2Keys/json, however, since both the tools 
> involve
>       outputting the entire content/keys of the SSTable in the order of the 
> keys,
>       they were not helpful in this case.
>       2. Since we wanted to get data on a few oldest cells with
>       timestamps, we created a tool mostly based off of sstable2json, called
>       sstableslicer, to output 'n' top/bottom cells in an SSTable, ordered 
> either
>       on writetime/localDeletionTime. This helped us identify the specific 
> cells
>       in new SSTables with older timestamps, which further helped in 
> debugging on
>       the application end. From application team perspective, however, writing
>       data with old timestamp is not a possible scenario.
>    3. Below is a sample output of sstableslicer
[image: Inline image 2]

> Looking for suggestions, especially around following two things:
>    1. Did we miss any other case in TWCS that could be causing such
>    overlap?
>    2. Does sstableslicer seem valuable, to be included in Apache C*? If
>    yes, I shall create a JIRA and submit a PR/patch for review.
> C* version we use is 2.1.17.

> Sumanth

Reply via email to