Fellow C* users, I've got a cluster of C* 3.5 serving a single keyspace with DTCS table and no deletes. We knew that data does not expire on time (even after gc_grace_period) - that's sth we wanted to investigate later and eventually let C* to keep it longer. The moment when C* decided to remove old data happened yesterday and on a one node we've reduced the data size by 40% (removing ~350 SSTables). On remaining nodes that was only ~5%. That's of course wrong and all nodes should get down by 40%. A closer look and we really found that ~40% is out-of-TTL and that maps to ~350 SSTables.
I've triggered a user defined compaction on few of the tables but ... nodetool compact
