> > Hi, > > We use TWCS in a few of the column families that have TTL based > time-series data, and no explicit deletes are issued. Over the time, we > observed the disk usage has been increasing beyond the expected levels. > > Data directory in a particular node shows SSTables that are more than > 16days old, while the bucket size is configured at 12hours, TTL is at > 15days and GC grace at 1hour. > Upon using sstableexpiredblockers, we got quite a few sets of blocking and > blocked SSTables. SSTableMetadata that is shown in the output indicates > there is an overlap in the MinTS-MaxTS period among the blocking SSTable > and the blocked SSTables, which is preventing the older SSTables from > getting dropped/deleted. > > Following are the possible root causes we considered > > 1. Hints - old data hints getting replayed from the coordinator node. > We ruled this out since hints live for no more than 1 day based on our > configuration. > 2. External compactions - no external compactions were run, that could > cause compaction of SSTables across the TWCS buckets. > 3. Read repairs - this is ruled out as well, since we never ran > external repairs, and read repair chance on the TWCS column families has > been set to 0. > 4. Application team writing data with older timestamp (in newer > SSTables). > > > 1. We wanted to identify the specific row keys with older timestamps > in the blocking SSTable, that could be causing this issue to occur. We > considered using SSTable2Keys/json, however, since both the tools > involve > outputting the entire content/keys of the SSTable in the order of the > keys, > they were not helpful in this case. > 2. Since we wanted to get data on a few oldest cells with > timestamps, we created a tool mostly based off of sstable2json, called > sstableslicer, to output 'n' top/bottom cells in an SSTable, ordered > either > on writetime/localDeletionTime. This helped us identify the specific > cells > in new SSTables with older timestamps, which further helped in > debugging on > the application end. From application team perspective, however, writing > data with old timestamp is not a possible scenario. > > 3. Below is a sample output of sstableslicer [image: Inline image 2]
> Looking for suggestions, especially around following two things: > > 1. Did we miss any other case in TWCS that could be causing such > overlap? > 2. Does sstableslicer seem valuable, to be included in Apache C*? If > yes, I shall create a JIRA and submit a PR/patch for review. > > C* version we use is 2.1.17. Thanks, > Sumanth >
