Re: Cassandra isn't compacting old files

These guesses will have to do. I thought something was wrong with such old SSTables. Thanks for your help investigating! On Wednesday, August 23, 2017, 3:09:34 AM PDT, kurt greaves wrote: Ignore me, I was getting the major compaction for LCS mixed up with STCS.

Re: Cassandra isn't compacting old files

Ignore me, I was getting the major compaction for LCS mixed up with STCS. Estimated droppable tombstones tends to be fairly accurate. If your SSTables in level 2 have that many tombstones I'd say that's definitely the reason L3 isn't being compacted. As for how you got here in the first place,

Re: Cassandra isn't compacting old files

I issued another major compaction just now and a brand new SSTable in Level 2 has an Estimated droppable tombstone value of 0.64. I don't know how accurate that is. On Tuesday, August 22, 2017, 9:33:34 PM PDT, Sotirios Delimanolis wrote: What do you mean by "a

Re: Cassandra isn't compacting old files

What do you mean by "a single SSTable"? SSTable size is set to 200MB and there are ~ 100 SSTables in that previous example in Level 3. This previous example table doesn't have a TTL, but we do delete rows. I've since compacted the table so I can't provide the previous "Estimated droppable

Re: Cassandra isn't compacting old files

LCS major compaction on 2.2 should compact each level to have a single SSTable. It seems more likely to me that you are simply not generating enough data to require compactions in L3 and most data is TTL'ing before it gets there. Out of curiosity, what does sstablemetadata report for Estimated

Re: Cassandra isn't compacting old files

Ignore the files missing those other components, that was confirmation bias :( I was sorting by date instead of by name and just assumed that something was wrong with Cassandra. Here's an example table's SSTables, sorted by level, then by repaired status: SSTable [name=lb-432055-big-Data.db,

Re: Cassandra isn't compacting old files

There seem to be a lot of SSTables in a repaired state and a lot in an unrepaired state. For example, for this one table, the logs report TRACE [main] 2017-08-15 23:50:30,732 LeveledManifest.java:473 - L0 contains 2 SSTables (176997267 bytes) in Manifest@1217144872 TRACE [main] 2017-08-15

Re: Cassandra isn't compacting old files

Turns out there are already logs for this in Tracker.java. I enabled those and clearly saw the old files are being tracked. What else can I look at for hints about whether these files are later invalidated/filtered out somehow? On Tuesday, August 1, 2017, 3:29:38 PM PDT, Sotirios Delimanolis

Re: Cassandra isn't compacting old files

There aren't any ERROR logs for failure to load these files and they do get compacted away. I'll try to plug some DEBUG logs in a custom Cassandra version.On Tuesday, August 1, 2017, 12:13:09 PM PDT, Jeff Jirsa wrote: I don't have time to dive deep into the code of your

Re: Cassandra isn't compacting old files

I don't have time to dive deep into the code of your version, but it may be ( https://issues.apache.org/jira/browse/CASSANDRA-13620 ) , or it may be something else. I wouldn't expect compaction to touch them if they're invalid. The handle may be a leftover from trying to load them. On Tue, Aug

Re: Cassandra isn't compacting old files

@Jeff, why does compaction clear them and why does Cassandra keep a handle to them? Shouldn't they be ignored entirely? Is there an error log I can enable to detect them? @kurt, there are no such logs for any of these tables. We have a custom log in our build of Cassandra that does shows that

Re: Cassandra isn't compacting old files

Seeing as there aren't even 100 SSTables in L2, LCS should be gradually trying to compact L3 with L2. You could search the logs for "Adding high-level (L3)" to check if this is happening. ​

Re: Cassandra isn't compacting old files

Yea, it means they're effecitvely invalid files, and would not be loaded at startup. On Mon, Jul 31, 2017 at 9:07 PM, Sotirios Delimanolis < sotodel...@yahoo.com.invalid> wrote: > I don't want to go down the TTL path because this behaviour is also > occurring for tables without a TTL. I don't

Re: Cassandra isn't compacting old files

I don't want to go down the TTL path because this behaviour is also occurring for tables without a TTL. I don't have hard numbers about the amount of writes, but there's definitely been enough to trigger compaction in the ~year since. We've never changed the topology of this cluster. Ranges have

Re: Cassandra isn't compacting old files

On 2017-07-31 15:00 (-0700), kurt greaves wrote: > How long is your ttl and how much data do you write per day (ie, what is > the difference in disk usage over a day)? Did you always TTL? > I'd say it's likely there is live data in those older sstables but you're > not

Re: Cassandra isn't compacting old files

How long is your ttl and how much data do you write per day (ie, what is the difference in disk usage over a day)? Did you always TTL? I'd say it's likely there is live data in those older sstables but you're not generating enough data to push new data to the highest level before it expires.

Cassandra isn't compacting old files

On Cassandra 2.2.11, I have a table that uses LeveledCompactionStrategy and that gets written to continuously. If I list the files in its data directory, I see something like this -rw-r--r-- 1 acassy agroup 161733811 Jul 31 18:46 lb-135346-big-Data.db -rw-r--r-- 1 acassy agroup 159626222 Jul 31