Re: Cassandra isn't compacting old files

[Sotirios Delimanolis]

These guesses will have to do. I thought something was wrong with such old 
SSTables.
Thanks for your help investigating!

On Wednesday, August 23, 2017, 3:09:34 AM PDT, kurt greaves 
 wrote:

Ignore me, I was getting the major compaction for LCS mixed up with STCS. 
Estimated droppable tombstones tends to be fairly accurate. If your SSTables in 
level 2 have that many tombstones I'd say that's definitely the reason L3 isn't 
being compacted.
As for how you got here in the first place, hard to say. Maybe a burst of 
writes over a period a long time ago? Or possibly repairs streaming a lot of 
data? Hard to tell what happened in the past without lots of metrics.​

Re: Cassandra isn't compacting old files

[kurt greaves]

Ignore me, I was getting the major compaction for LCS mixed up with STCS.
Estimated droppable tombstones tends to be fairly accurate. If your
SSTables in level 2 have that many tombstones I'd say that's definitely the
reason L3 isn't being compacted.

As for how you got here in the first place, hard to say. Maybe a burst of
writes over a period a long time ago? Or possibly repairs streaming a lot
of data? Hard to tell what happened in the past without lots of metrics.
​

Re: Cassandra isn't compacting old files

[Sotirios Delimanolis]

I issued another major compaction just now and a brand new SSTable in Level 2 
has an Estimated droppable tombstone value of 0.64. I don't know how accurate 
that is.

On Tuesday, August 22, 2017, 9:33:34 PM PDT, Sotirios Delimanolis 
 wrote:

What do you mean by "a single SSTable"? SSTable size is set to 200MB and there 
are ~ 100 SSTables in that previous example in Level 3.
This previous example table doesn't have a TTL, but we do delete rows. I've 
since compacted the table so I can't provide the previous "Estimated droppable 
tombstones", but it was > 0.3. I've set the threshold to 0.25, but 
unchecked_tombstone_compaction is false. Perhaps setting it to true would 
eventually compact individual SSTables.
I agree that I am probably not creating enough data at the moment, but what got 
me into this situation in the first place? All (+/- a couple) SSTables in each 
level are last modified on the same date. 
On Tuesday, August 22, 2017, 5:16:27 PM PDT, kurt greaves 
 wrote:

LCS major compaction on 2.2 should compact each level to have a single SSTable. 
It seems more likely to me that you are simply not generating enough data to 
require compactions in L3 and most data is TTL'ing before it gets there. Out of 
curiosity, what does sstablemetadata report for  Estimated droppable tombstones 
on one of those tables, and what is your TTL?​

Re: Cassandra isn't compacting old files

[Sotirios Delimanolis]

What do you mean by "a single SSTable"? SSTable size is set to 200MB and there 
are ~ 100 SSTables in that previous example in Level 3.
This previous example table doesn't have a TTL, but we do delete rows. I've 
since compacted the table so I can't provide the previous "Estimated droppable 
tombstones", but it was > 0.3. I've set the threshold to 0.25, but 
unchecked_tombstone_compaction is false. Perhaps setting it to true would 
eventually compact individual SSTables.
I agree that I am probably not creating enough data at the moment, but what got 
me into this situation in the first place? All (+/- a couple) SSTables in each 
level are last modified on the same date. 
On Tuesday, August 22, 2017, 5:16:27 PM PDT, kurt greaves 
 wrote:

LCS major compaction on 2.2 should compact each level to have a single SSTable. 
It seems more likely to me that you are simply not generating enough data to 
require compactions in L3 and most data is TTL'ing before it gets there. Out of 
curiosity, what does sstablemetadata report for  Estimated droppable tombstones 
on one of those tables, and what is your TTL?​

Re: Cassandra isn't compacting old files

[kurt greaves]

LCS major compaction on 2.2 should compact each level to have a single
SSTable. It seems more likely to me that you are simply not generating
enough data to require compactions in L3 and most data is TTL'ing before it
gets there. Out of curiosity, what does sstablemetadata report for
 Estimated droppable tombstones on one of those tables, and what is your
TTL?​

Re: Cassandra isn't compacting old files

[Sotirios Delimanolis]

Ignore the files missing those other components, that was confirmation bias :( 
I was sorting by date instead of by name and just assumed that something was 
wrong with Cassandra.
Here's an example table's SSTables, sorted by level, then by repaired status:
SSTable [name=lb-432055-big-Data.db, level=0, repaired=false, 
instant=2017-08-22] Level 1 SSTable 
[name=lb-431497-big-Data.db, level=1, repaired=false, instant=2017-08-17]
SSTable [name=lb-431496-big-Data.db, level=1, repaired=false, 
instant=2017-08-17]
SSTable [name=lb-431495-big-Data.db, level=1, repaired=false, 
instant=2017-08-17]
SSTable [name=lb-431498-big-Data.db, level=1, repaired=false, 
instant=2017-08-17]
SSTable [name=lb-431499-big-Data.db, level=1, repaired=false, 
instant=2017-08-17]
SSTable [name=lb-431501-big-Data.db, level=1, repaired=false, 
instant=2017-08-17]
SSTable [name=lb-431503-big-Data.db, level=1, repaired=false, 
instant=2017-08-17]
SSTable [name=lb-431500-big-Data.db, level=1, repaired=false, 
instant=2017-08-17]
SSTable [name=lb-431502-big-Data.db, level=1, repaired=false, 
instant=2017-08-17]
SSTable [name=lb-431504-big-Data.db, level=1, repaired=false, 
instant=2017-08-17]
SSTable [name=lb-426107-big-Data.db, level=1, repaired=true, instant=2017-07-07]
SSTable [name=lb-426105-big-Data.db, level=1, repaired=true, instant=2017-07-07]
SSTable [name=lb-426090-big-Data.db, level=1, repaired=true, instant=2017-07-07]
SSTable [name=lb-426092-big-Data.db, level=1, repaired=true, instant=2017-07-07]
SSTable [name=lb-426094-big-Data.db, level=1, repaired=true, instant=2017-07-07]
SSTable [name=lb-426096-big-Data.db, level=1, repaired=true, instant=2017-07-07]
SSTable [name=lb-426104-big-Data.db, level=1, repaired=true, instant=2017-07-07]
SSTable [name=lb-426102-big-Data.db, level=1, repaired=true, instant=2017-07-07]
SSTable [name=lb-426100-big-Data.db, level=1, repaired=true, 
instant=2017-07-07] Level 2 SSTable 
[name=lb-423829-big-Data.db, level=2, repaired=false, instant=2017-06-23]
SSTable [name=lb-431505-big-Data.db, level=2, repaired=false, 
instant=2017-08-17]
SSTable [name=lb-423830-big-Data.db, level=2, repaired=false, 
instant=2017-06-23]
SSTable [name=lb-424559-big-Data.db, level=2, repaired=false, 
instant=2017-06-29]
SSTable [name=lb-424568-big-Data.db, level=2, repaired=false, 
instant=2017-06-29]
SSTable [name=lb-424567-big-Data.db, level=2, repaired=false, 
instant=2017-06-29]
SSTable [name=lb-424566-big-Data.db, level=2, repaired=false, 
instant=2017-06-29]
SSTable [name=lb-424561-big-Data.db, level=2, repaired=false, 
instant=2017-06-29]
SSTable [name=lb-424563-big-Data.db, level=2, repaired=false, 
instant=2017-06-29]
SSTable [name=lb-424562-big-Data.db, level=2, repaired=false, 
instant=2017-06-29]
SSTable [name=lb-423825-big-Data.db, level=2, repaired=false, 
instant=2017-06-23]
SSTable [name=lb-423823-big-Data.db, level=2, repaired=false, 
instant=2017-06-23]
SSTable [name=lb-424560-big-Data.db, level=2, repaired=false, 
instant=2017-06-29]
SSTable [name=lb-423824-big-Data.db, level=2, repaired=false, 
instant=2017-06-23]
SSTable [name=lb-423828-big-Data.db, level=2, repaired=false, 
instant=2017-06-23]
SSTable [name=lb-423826-big-Data.db, level=2, repaired=false, 
instant=2017-06-23]
SSTable [name=lb-423380-big-Data.db, level=2, repaired=false, 
instant=2017-06-20]
SSTable [name=lb-426057-big-Data.db, level=2, repaired=true, instant=2017-07-07]
SSTable [name=lb-426058-big-Data.db, level=2, repaired=true, 
instant=2017-07-07][...~60 more from 2017-07-07...]
SSTable [name=lb-425991-big-Data.db, level=2, repaired=true, instant=2017-07-07]
SSTable [name=lb-426084-big-Data.db, level=2, repaired=true, 
instant=2017-07-07] Level 3 SSTable 
[name=lb-383142-big-Data.db, level=3, repaired=false, instant=2016-11-19]
SSTable [name=lb-383143-big-Data.db, level=3, repaired=false, 
instant=2016-11-19][...~40 more from 2016-11-19...]
SSTable [name=lb-383178-big-Data.db, level=3, repaired=false, 
instant=2016-11-19]
SSTable [name=lb-383188-big-Data.db, level=3, repaired=false, 
instant=2016-11-19]
SSTable [name=lb-425948-big-Data.db, level=3, repaired=false, 
instant=2017-07-07]
SSTable [name=lb-383179-big-Data.db, level=3, repaired=false, 
instant=2016-11-19]
SSTable [name=lb-383175-big-Data.db, level=3, repaired=false, 
instant=2016-11-19][...~30 more from 2016-11-19...]
SSTable [name=lb-383160-big-Data.db, level=3, repaired=false, 
instant=2016-11-19]
SSTable [name=lb-383181-big-Data.db, level=3, repaired=false, 
instant=2016-11-19]
SSTable [name=lb-383258-big-Data.db, level=3, repaired=true, instant=2016-11-19]
SSTable [name=lb-383256-big-Data.db, level=3, repaired=true, instant=2016-11-19]
SSTable [name=lb-386829-big-Data.db, level=3, repaired=true, instant=2016-11-30]
SSTable [name=lb-383251-big-Data.db, level=3, repaired=true, instant=2016-11-19]
SSTable [name=lb-383259-big-Data.db, level=3, repaired=true, instant=2016-11-19]
SSTable 

Re: Cassandra isn't compacting old files

[Sotirios Delimanolis]

There seem to be a lot of SSTables in a repaired state and a lot in an 
unrepaired state. For example, for this one table, the logs report

TRACE [main] 2017-08-15 23:50:30,732 LeveledManifest.java:473 - L0 contains 2 
SSTables (176997267 bytes) in Manifest@1217144872
TRACE [main] 2017-08-15 23:50:30,732 LeveledManifest.java:473 - L1 contains 10 
SSTables (2030691642 bytes) in Manifest@1217144872
TRACE [main] 2017-08-15 23:50:30,732 LeveledManifest.java:473 - L2 contains 94 
SSTables (19352545435 bytes) in Manifest@1217144872

and 

TRACE [main] 2017-08-15 23:50:30,731 LeveledManifest.java:473 - L0 contains 1 
SSTables (65038718 bytes) in Manifest@499561185
TRACE [main] 2017-08-15 23:50:30,731 LeveledManifest.java:473 - L2 contains 5 
SSTables (11722 bytes) in Manifest@499561185
TRACE [main] 2017-08-15 23:50:30,731 LeveledManifest.java:473 - L3 contains 39 
SSTables (7377654173 bytes) in Manifest@499561185

Is it possible that there's always a compaction to be run in the "repaired" 
state, with that many SSTables, that unrepaired compactions are essentially 
"starved", considering the WrappingCompactionStrategy prioritizes the 
"repaired" set?On Wednesday, August 2, 2017, 2:35:02 PM PDT, Sotirios 
Delimanolis  wrote:

Turns out there are already logs for this in Tracker.java. I enabled those and 
clearly saw the old files are being tracked.
What else can I look at for hints about whether these files are later 
invalidated/filtered out somehow?

On Tuesday, August 1, 2017, 3:29:38 PM PDT, Sotirios Delimanolis 
 wrote:

There aren't any ERROR logs for failure to load these files and they do get 
compacted away. I'll try to plug some DEBUG logs in a custom Cassandra 
version.On Tuesday, August 1, 2017, 12:13:09 PM PDT, Jeff Jirsa 
 wrote:

I don't have time to dive deep into the code of your version, but it may be ( 
https://issues.apache.org/jira/browse/CASSANDRA-13620 ) , or it may be 
something else.
I wouldn't expect compaction to touch them if they're invalid. The handle may 
be a leftover from trying to load them. 


On Tue, Aug 1, 2017 at 10:01 AM, Sotirios Delimanolis 
 wrote:

@Jeff, why does compaction clear them and why does Cassandra keep a handle to 
them? Shouldn't they be ignored entirely? Is there an error log I can enable to 
detect them?
@kurt, there are no such logs for any of these tables. We have a custom log in 
our build of Cassandra that does shows that compactions are happening for that 
table but only ever include the files from July.
On Tuesday, August 1, 2017, 12:55:53 AM PDT, kurt greaves 
 wrote:

Seeing as there aren't even 100 SSTables in L2, LCS should be gradually trying 
to compact L3 with L2. You could search the logs for "Adding high-level (L3)" 
to check if this is happening. ​

Re: Cassandra isn't compacting old files

[Sotirios Delimanolis]

Turns out there are already logs for this in Tracker.java. I enabled those and 
clearly saw the old files are being tracked.
What else can I look at for hints about whether these files are later 
invalidated/filtered out somehow?

On Tuesday, August 1, 2017, 3:29:38 PM PDT, Sotirios Delimanolis 
 wrote:

There aren't any ERROR logs for failure to load these files and they do get 
compacted away. I'll try to plug some DEBUG logs in a custom Cassandra 
version.On Tuesday, August 1, 2017, 12:13:09 PM PDT, Jeff Jirsa 
 wrote:

I don't have time to dive deep into the code of your version, but it may be ( 
https://issues.apache.org/jira/browse/CASSANDRA-13620 ) , or it may be 
something else.
I wouldn't expect compaction to touch them if they're invalid. The handle may 
be a leftover from trying to load them. 


On Tue, Aug 1, 2017 at 10:01 AM, Sotirios Delimanolis 
 wrote:

@Jeff, why does compaction clear them and why does Cassandra keep a handle to 
them? Shouldn't they be ignored entirely? Is there an error log I can enable to 
detect them?
@kurt, there are no such logs for any of these tables. We have a custom log in 
our build of Cassandra that does shows that compactions are happening for that 
table but only ever include the files from July.
On Tuesday, August 1, 2017, 12:55:53 AM PDT, kurt greaves 
 wrote:

Seeing as there aren't even 100 SSTables in L2, LCS should be gradually trying 
to compact L3 with L2. You could search the logs for "Adding high-level (L3)" 
to check if this is happening. ​

Re: Cassandra isn't compacting old files

[Sotirios Delimanolis]

There aren't any ERROR logs for failure to load these files and they do get 
compacted away. I'll try to plug some DEBUG logs in a custom Cassandra 
version.On Tuesday, August 1, 2017, 12:13:09 PM PDT, Jeff Jirsa 
 wrote:

I don't have time to dive deep into the code of your version, but it may be ( 
https://issues.apache.org/jira/browse/CASSANDRA-13620 ) , or it may be 
something else.
I wouldn't expect compaction to touch them if they're invalid. The handle may 
be a leftover from trying to load them. 


On Tue, Aug 1, 2017 at 10:01 AM, Sotirios Delimanolis 
 wrote:

@Jeff, why does compaction clear them and why does Cassandra keep a handle to 
them? Shouldn't they be ignored entirely? Is there an error log I can enable to 
detect them?
@kurt, there are no such logs for any of these tables. We have a custom log in 
our build of Cassandra that does shows that compactions are happening for that 
table but only ever include the files from July.
On Tuesday, August 1, 2017, 12:55:53 AM PDT, kurt greaves 
 wrote:

Seeing as there aren't even 100 SSTables in L2, LCS should be gradually trying 
to compact L3 with L2. You could search the logs for "Adding high-level (L3)" 
to check if this is happening. ​

Re: Cassandra isn't compacting old files

[Jeff Jirsa]

I don't have time to dive deep into the code of your version, but it may be
( https://issues.apache.org/jira/browse/CASSANDRA-13620 ) , or it may be
something else.

I wouldn't expect compaction to touch them if they're invalid. The handle
may be a leftover from trying to load them.



On Tue, Aug 1, 2017 at 10:01 AM, Sotirios Delimanolis <
sotodel...@yahoo.com.invalid> wrote:

> @Jeff, why does compaction clear them and why does Cassandra keep a handle
> to them? Shouldn't they be ignored entirely? Is there an error log I can
> enable to detect them?
>
> @kurt, there are no such logs for any of these tables. We have a custom
> log in our build of Cassandra that does shows that compactions are
> happening for that table but only ever include the files from July.
>
> On Tuesday, August 1, 2017, 12:55:53 AM PDT, kurt greaves <
> k...@instaclustr.com> wrote:
>
>
> Seeing as there aren't even 100 SSTables in L2, LCS should be gradually
> trying to compact L3 with L2. You could search the logs for "Adding
> high-level (L3)" to check if this is happening. ​
>

Re: Cassandra isn't compacting old files

[Sotirios Delimanolis]

@Jeff, why does compaction clear them and why does Cassandra keep a handle to 
them? Shouldn't they be ignored entirely? Is there an error log I can enable to 
detect them?
@kurt, there are no such logs for any of these tables. We have a custom log in 
our build of Cassandra that does shows that compactions are happening for that 
table but only ever include the files from July.
On Tuesday, August 1, 2017, 12:55:53 AM PDT, kurt greaves 
 wrote:

Seeing as there aren't even 100 SSTables in L2, LCS should be gradually trying 
to compact L3 with L2. You could search the logs for "Adding high-level (L3)" 
to check if this is happening. ​

Re: Cassandra isn't compacting old files

[kurt greaves]

Seeing as there aren't even 100 SSTables in L2, LCS should be gradually
trying to compact L3 with L2. You could search the logs for "Adding
high-level (L3)" to check if this is happening. ​

Re: Cassandra isn't compacting old files

[Jeff Jirsa]

Yea, it means they're effecitvely invalid files, and would not be loaded at
startup.



On Mon, Jul 31, 2017 at 9:07 PM, Sotirios Delimanolis <
sotodel...@yahoo.com.invalid> wrote:

> I don't want to go down the TTL path because this behaviour is also
> occurring for tables without a TTL. I don't have hard numbers about the
> amount of writes, but there's definitely been enough to trigger compaction
> in the ~year since.
>
> We've never changed the topology of this cluster. Ranges have always been
> the same.
>
> I can't remember about repairs, but running sstablemetadata shows
>
> Repaired at: 0
>
> across all files. The Cassandra process has been restarted multiple times
> in the last year.
>
> I'v noticed there are only -Data.db and -Index.db files in some rare
> cases. The compression info, filter, summary, and statistics files are
> missing. Does that hint at anything?
>
>
>
>
> On Monday, July 31, 2017, 3:39:11 PM PDT, Jeff Jirsa 
> wrote:
>
>
>
>
> On 2017-07-31 15:00 (-0700), kurt greaves  wrote:
> > How long is your ttl and how much data do you write per day (ie, what is
> > the difference in disk usage over a day)? Did you always TTL?
> > I'd say it's likely there is live data in those older sstables but you're
> > not generating enough data to push new data to the highest level before
> it
> > expires.
>
> >
>
> This is a pretty good option. Other options:
>
> 1) You changed topology on Nov 28, and the ranges covered by those
> sstables are no longer intersecting with the ranges on the node, so they're
> not being selected as LCS compaction candidates (and if you run nodetool
> cleanup, they probably get deleted)
>
> 2) You ran incremental repairs once, and stopped on the 28th, and now
> those sstables have a repairedAt set, so they won't be compacted with other
> (unrepaired) sstables
>
> 3) There's some horrible bug where the sstables got lost from the running
> daemon, and if you restart it'll magically get sucked in and start working
> again (this is really unlikely, and it would be a very bad bug).
>
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>
>

Re: Cassandra isn't compacting old files

[Sotirios Delimanolis]

I don't want to go down the TTL path because this behaviour is also occurring 
for tables without a TTL. I don't have hard numbers about the amount of writes, 
but there's definitely been enough to trigger compaction in the ~year since.
We've never changed the topology of this cluster. Ranges have always been the 
same. 
I can't remember about repairs, but running sstablemetadata shows
Repaired at: 0
across all files. The Cassandra process has been restarted multiple times in 
the last year.
I'v noticed there are only -Data.db and -Index.db files in some rare cases. The 
compression info, filter, summary, and statistics files are missing. Does that 
hint at anything?




On Monday, July 31, 2017, 3:39:11 PM PDT, Jeff Jirsa  wrote:



On 2017-07-31 15:00 (-0700), kurt greaves  wrote: 
> How long is your ttl and how much data do you write per day (ie, what is
> the difference in disk usage over a day)? Did you always TTL?
> I'd say it's likely there is live data in those older sstables but you're
> not generating enough data to push new data to the highest level before it
> expires.
> 

This is a pretty good option. Other options:

1) You changed topology on Nov 28, and the ranges covered by those sstables are 
no longer intersecting with the ranges on the node, so they're not being 
selected as LCS compaction candidates (and if you run nodetool cleanup, they 
probably get deleted)

2) You ran incremental repairs once, and stopped on the 28th, and now those 
sstables have a repairedAt set, so they won't be compacted with other 
(unrepaired) sstables

3) There's some horrible bug where the sstables got lost from the running 
daemon, and if you restart it'll magically get sucked in and start working 
again (this is really unlikely, and it would be a very bad bug).



-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Cassandra isn't compacting old files

[Jeff Jirsa]

On 2017-07-31 15:00 (-0700), kurt greaves  wrote: 
> How long is your ttl and how much data do you write per day (ie, what is
> the difference in disk usage over a day)? Did you always TTL?
> I'd say it's likely there is live data in those older sstables but you're
> not generating enough data to push new data to the highest level before it
> expires.
> 

This is a pretty good option. Other options:

1) You changed topology on Nov 28, and the ranges covered by those sstables are 
no longer intersecting with the ranges on the node, so they're not being 
selected as LCS compaction candidates (and if you run nodetool cleanup, they 
probably get deleted)

2) You ran incremental repairs once, and stopped on the 28th, and now those 
sstables have a repairedAt set, so they won't be compacted with other 
(unrepaired) sstables

3) There's some horrible bug where the sstables got lost from the running 
daemon, and if you restart it'll magically get sucked in and start working 
again (this is really unlikely, and it would be a very bad bug).



-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Cassandra isn't compacting old files

[kurt greaves]

How long is your ttl and how much data do you write per day (ie, what is
the difference in disk usage over a day)? Did you always TTL?
I'd say it's likely there is live data in those older sstables but you're
not generating enough data to push new data to the highest level before it
expires.