Re: TWCS sstables not dropping even though all data is expired

Mike, It has taken me sometime, but I have now written this up in more detail on my blog: http://www.redshots.com/cassandra-twcs-must-have-ttls/ However I couldn’t get the tombstone compaction subproperties to work as expected. If I

Re: TWCS sstables not dropping even though all data is expired

ll, share your compaction settings. More than likely, your issue >> is from 1 of 2 reasons: >> 1. You have read repair chance set to anything other than 0 >> >> 2. You’re running repairs on the TWCS CF >> >> >> >> Or both…. >> &

Re: TWCS sstables not dropping even though all data is expired

repairs on the TWCS CF >>> >>> >>> >>> Or both…. >>> >>> >>> >>> From: Mike Torra [mailto:mto...@salesforce.com.INVALID] >>> Sent: Friday, May 03, 2019 3:00 PM >>> To: user@cassandra.apache.org &

Re: TWCS sstables not dropping even though all data is expired

epairs on the TWCS CF > > > > Or both…. > > > > *From:* Mike Torra [mailto:mto...@salesforce.com.INVALID > ] > *Sent:* Friday, May 03, 2019 3:00 PM > *To:* user@cassandra.apache.org > *Subject:* Re: TWCS sstables not dropping even though all data is expired &

Re: TWCS sstables not dropping even though all data is expired

both…. > > From: Mike Torra [mailto:mto...@salesforce.com.INVALID] > Sent: Friday, May 03, 2019 3:00 PM > To: user@cassandra.apache.org > Subject: Re: TWCS sstables not dropping even though all data is expired > > Thx for the help Paul - there are definitely some details here I

RE: TWCS sstables not dropping even though all data is expired

, 2019 3:00 PM To: user@cassandra.apache.org Subject: Re: TWCS sstables not dropping even though all data is expired Thx for the help Paul - there are definitely some details here I still don't fully understand, but this helped me resolve the problem and know what to look for in the future

Re: TWCS sstables not dropping even though all data is expired

Thx for the help Paul - there are definitely some details here I still don't fully understand, but this helped me resolve the problem and know what to look for in the future :) On Fri, May 3, 2019 at 12:44 PM Paul Chandler wrote: > Hi Mike, > > For TWCS the sstable can only be deleted when all

Re: TWCS sstables not dropping even though all data is expired

Hi Mike, For TWCS the sstable can only be deleted when all the data has expired in that sstable, but you had a record without a ttl in it, so that sstable could never be deleted. That bit is straight forward, the next bit I remember reading somewhere but can’t find it at the moment to confirm

Re: TWCS sstables not dropping even though all data is expired

This does indeed seem to be a problem of overlapping sstables, but I don't understand why the data (and number of sstables) just continues to grow indefinitely. I also don't understand why this problem is only appearing on some nodes. Is it just a coincidence that the one rogue test row without a

Re: TWCS sstables not dropping even though all data is expired

Hi Mike, It sounds like that record may have been deleted, if that is the case then it would still be shown in this sstable, but the deleted tombstone record would be in a later sstable. You can use nodetool getsstables to work out which sstables contain the data. I recommend reading The Last

RE: TWCS sstables not dropping even though all data is expired

Hi Mike, Have you checked to make sure you’re not a victim of timestamp overlap? From: Mike Torra [mailto:mto...@salesforce.com.INVALID] Sent: Thursday, May 02, 2019 11:09 AM To: user@cassandra.apache.org Subject: Re: TWCS sstables not dropping even though all data is expired I'm pretty stumped

Re: TWCS sstables not dropping even though all data is expired

I'm pretty stumped by this, so here is some more detail if it helps. Here is what the suspicious partition looks like in the `sstabledump` output (some pii etc redacted): ``` { "partition" : { "key" : [ "some_user_id_value", "user_id", "demo-test" ], "position" : 210 },

TWCS sstables not dropping even though all data is expired

Hello - I have a 48 node C* cluster spread across 4 AWS regions with RF=3. A few months ago I started noticing disk usage on some nodes increasing consistently. At first I solved the problem by destroying the nodes and rebuilding them, but the problem returns. I did some more investigation