I have a database which has a fair amount of churn. When I need to update a
data structure, I create a new one, and when it is complete, I delete the old
one. I have gc_grace_seconds=0, so the space for the old data structures should
be reclaimed on the next compaction. This has been working fine for months.
Unfortunately, compaction has stopped doing its job, and about a week ago disk
space started increasing steadily. Disk space usage has increased almost 50% in
the last week.
I do see compaction tasks get kicked off, but they complete very quickly,
generally in well less than a minute. Compaction of the bigger tables used to
take more than an hour. Admittedly, its been quite a while since I’ve watched
compactions run, and I probably haven’t watched them since we last upgraded, so
perhaps there’s been a change in how LeveledCompaction works. Our last upgrade
was well more than a week ago, so an upgrade isn’t responsible for this change
in behavior. We’re running 2.0.14 currently.
Looking at the logs, I see stuff like this:
INFO [CompactionExecutor:49180] 2015-07-24 09:31:45,960 CompactionTask.java
(line 296) Compacted 2 sstables to
[/var/lib/cassandra/data/fold31_browse/node/fold31_browse-node-jb-11110,].
117,066,921 bytes to 117,346,402 (~100% of original) in 33,766ms =
3.314288MB/s. 1,444,907 total partitions merged to 1,400,563. Partition merge
counts were {1:1356219, 2:44344, }
So, compaction seems to be doing something, just not very much. I’ve checked
the process that deletes stale data, and it is doing its job. Plenty of
deletion going on, just no space reclamation.
Any thoughts?
Thanks in advance
Robert
