Fwiw if you enable the tombstone compaction subproperties, you’ll compact away
most of the other data in those old sstables (but not the partition that’s been
manually updated)
Also table level TTLs help catch this type of manual manipulation - consider
adding it if appropriate.
--
Jeff Jirsa
> On May 6, 2019, at 7:29 AM, Mike Torra <mto...@salesforce.com.invalid> wrote:
>
> Compaction settings:
> ```
> compaction = {'class':
> 'org.apache.cassandra.db.compaction.TimeWindowCompactionStrategy',
> 'compaction_window_size': '6', 'compaction_window_unit': 'HOURS',
> 'max_threshold': '32', 'min_threshold': '4'}
> ```
> read_repair_chance is 0, and I don't do any repairs because (normally)
> everything has a ttl. It does seem like Jeff is right that a manual
> insert/update without a ttl is what caused this, so I know how to resolve it
> and prevent it from happening again.
>
> Thx again for all the help guys, I appreciate it!
>
>
>> On Fri, May 3, 2019 at 11:21 PM Jeff Jirsa <jji...@gmail.com> wrote:
>> Repairs work fine with TWCS, but having a non-expiring row will prevent
>> tombstones in newer sstables from being purged
>>
>> I suspect someone did a manual insert/update without a ttl and that
>> effectively blocks all other expiring cells from being purged.
>>
>> --
>> Jeff Jirsa
>>
>>
>>> On May 3, 2019, at 7:57 PM, Nick Hatfield <nick.hatfi...@metricly.com>
>>> wrote:
>>>
>>> Hi Mike,
>>>
>>>
>>>
>>> If you will, share your compaction settings. More than likely, your issue
>>> is from 1 of 2 reasons:
>>> 1. You have read repair chance set to anything other than 0
>>>
>>> 2. You’re running repairs on the TWCS CF
>>>
>>>
>>>
>>> Or both….
>>>
>>>
>>>
>>> From: Mike Torra [mailto:mto...@salesforce.com.INVALID]
>>> Sent: Friday, May 03, 2019 3:00 PM
>>> To: user@cassandra.apache.org
>>> Subject: Re: TWCS sstables not dropping even though all data is expired
>>>
>>>
>>>
>>> Thx for the help Paul - there are definitely some details here I still
>>> don't fully understand, but this helped me resolve the problem and know
>>> what to look for in the future :)
>>>
>>>
>>>
>>> On Fri, May 3, 2019 at 12:44 PM Paul Chandler <p...@redshots.com> wrote:
>>>
>>> Hi Mike,
>>>
>>>
>>>
>>> For TWCS the sstable can only be deleted when all the data has expired in
>>> that sstable, but you had a record without a ttl in it, so that sstable
>>> could never be deleted.
>>>
>>>
>>>
>>> That bit is straight forward, the next bit I remember reading somewhere but
>>> can’t find it at the moment to confirm my thinking.
>>>
>>>
>>>
>>> An sstable can only be deleted if it is the earliest sstable. I think this
>>> is due to the fact that deleting later sstables may expose old versions of
>>> the data stored in the stuck sstable which had been superseded. For
>>> example, if there was a tombstone in a later sstable for the non TTLed
>>> record causing the problem in this instance. Then deleting that sstable
>>> would cause that deleted data to reappear. (Someone please correct me if I
>>> have this wrong)
>>>
>>>
>>>
>>> Because sstables in different time buckets are never compacted together,
>>> this problem only goes away when you did the major compaction.
>>>
>>>
>>>
>>> This would happen on all replicas of the data, hence the reason you this
>>> problem on 3 nodes.
>>>
>>>
>>>
>>> Thanks
>>>
>>>
>>>
>>> Paul
>>>
>>> www.redshots.com
>>>
>>>
>>>
>>>
>>> On 3 May 2019, at 15:35, Mike Torra <mto...@salesforce.com.INVALID> wrote:
>>>
>>>
>>>
>>> This does indeed seem to be a problem of overlapping sstables, but I don't
>>> understand why the data (and number of sstables) just continues to grow
>>> indefinitely. I also don't understand why this problem is only appearing on
>>> some nodes. Is it just a coincidence that the one rogue test row without a
>>> ttl is at the 'root' sstable causing the problem (ie, from the output of
>>> `sstableexpiredblockers`)?
>>>
>>>
>>>
>>> Running a full compaction via `nodetool compact` reclaims the disk space,
>>> but I'd like to figure out why this happened and prevent it. Understanding
>>> why this problem would be isolated the way it is (ie only one CF even
>>> though I have a few others that share a very similar schema, and only some
>>> nodes) seems like it will help me prevent it.
>>>
>>>
>>>
>>>
>>>
>>> On Thu, May 2, 2019 at 1:00 PM Paul Chandler <p...@redshots.com> wrote:
>>>
>>> Hi Mike,
>>>
>>>
>>>
>>> It sounds like that record may have been deleted, if that is the case then
>>> it would still be shown in this sstable, but the deleted tombstone record
>>> would be in a later sstable. You can use nodetool getsstables to work out
>>> which sstables contain the data.
>>>
>>>
>>>
>>> I recommend reading The Last Pickle post on this:
>>> http://thelastpickle.com/blog/2016/12/08/TWCS-part1.html the sections
>>> towards the bottom of this post may well explain why the sstable is not
>>> being deleted.
>>>
>>>
>>>
>>> Thanks
>>>
>>>
>>>
>>> Paul
>>>
>>> www.redshots.com
>>>
>>>
>>>
>>>
>>> On 2 May 2019, at 16:08, Mike Torra <mto...@salesforce.com.INVALID> wrote:
>>>
>>>
>>>
>>> I'm pretty stumped by this, so here is some more detail if it helps.
>>>
>>>
>>>
>>> Here is what the suspicious partition looks like in the `sstabledump`
>>> output (some pii etc redacted):
>>>
>>> ```
>>>
>>> {
>>>
>>> "partition" : {
>>>
>>> "key" : [ "some_user_id_value", "user_id", "demo-test" ],
>>>
>>> "position" : 210
>>>
>>> },
>>>
>>> "rows" : [
>>>
>>> {
>>>
>>> "type" : "row",
>>>
>>> "position" : 1132,
>>>
>>> "clustering" : [ "2019-01-22 15:27:45.000Z" ],
>>>
>>> "liveness_info" : { "tstamp" : "2019-01-22T15:31:12.415081Z" },
>>>
>>> "cells" : [
>>>
>>> { "some": "data" }
>>>
>>> ]
>>>
>>> }
>>>
>>> ]
>>>
>>> }
>>>
>>> ```
>>>
>>>
>>>
>>> And here is what every other partition looks like:
>>>
>>> ```
>>>
>>> {
>>>
>>> "partition" : {
>>>
>>> "key" : [ "some_other_user_id", "user_id", "some_site_id" ],
>>>
>>> "position" : 1133
>>>
>>> },
>>>
>>> "rows" : [
>>>
>>> {
>>>
>>> "type" : "row",
>>>
>>> "position" : 1234,
>>>
>>> "clustering" : [ "2019-01-22 17:59:35.547Z" ],
>>>
>>> "liveness_info" : { "tstamp" : "2019-01-22T17:59:35.708Z", "ttl" :
>>> 86400, "expires_at" : "2019-01-23T17:59:35Z", "expired" : true },
>>>
>>> "cells" : [
>>>
>>> { "name" : "activity_data", "deletion_info" : {
>>> "local_delete_time" : "2019-01-22T17:59:35Z" }
>>>
>>> }
>>>
>>> ]
>>>
>>> }
>>>
>>> ]
>>>
>>> }
>>>
>>> ```
>>>
>>>
>>>
>>> As expected, almost all of the data except this one suspicious partition
>>> has a ttl and is already expired. But if a partition isn't expired and I
>>> see it in the sstable, why wouldn't I see it executing a CQL query against
>>> the CF? Why would this sstable be preventing so many other sstable's from
>>> getting cleaned up?
>>>
>>>
>>>
>>> On Tue, Apr 30, 2019 at 12:34 PM Mike Torra <mto...@salesforce.com> wrote:
>>>
>>> Hello -
>>>
>>>
>>>
>>> I have a 48 node C* cluster spread across 4 AWS regions with RF=3. A few
>>> months ago I started noticing disk usage on some nodes increasing
>>> consistently. At first I solved the problem by destroying the nodes and
>>> rebuilding them, but the problem returns.
>>>
>>>
>>>
>>> I did some more investigation recently, and this is what I found:
>>>
>>> - I narrowed the problem down to a CF that uses TWCS, by simply looking at
>>> disk space usage
>>>
>>> - in each region, 3 nodes have this problem of growing disk space (matches
>>> replication factor)
>>>
>>> - on each node, I tracked down the problem to a particular SSTable using
>>> `sstableexpiredblockers`
>>>
>>> - in the SSTable, using `sstabledump`, I found a row that does not have a
>>> ttl like the other rows, and appears to be from someone else on the team
>>> testing something and forgetting to include a ttl
>>>
>>> - all other rows show "expired: true" except this one, hence my suspicion
>>>
>>> - when I query for that particular partition key, I get no results
>>>
>>> - I tried deleting the row anyways, but that didn't seem to change anything
>>>
>>> - I also tried `nodetool scrub`, but that didn't help either
>>>
>>>
>>>
>>> Would this rogue row without a ttl explain the problem? If so, why? If not,
>>> does anyone have any other ideas? Why does the row show in `sstabledump`
>>> but not when I query for it?
>>>
>>>
>>>
>>> I appreciate any help or suggestions!
>>>
>>>
>>>
>>> - Mike
>>>
>>>
>>>
>>>
