Hey
I have a 4.0.7 cluster in which I see weird behavior.
I expect that once compaction finishes, the old sstables that were part of
the compaction set will be deleted, but it appears they are deleted much
later, thus causing space issues.
For example this is what I have in the log, only one compaction finished,
which took as input 3 sstables, but I can see many sstables were deleted
afterwards:
INFO [CompactionExecutor:17344] 2023-04-04 04:39:07,345
CompactionTask.java:241 - Compacted (489ddc80-d282-11ed-b4fb-d9d812cda140)
3 sstables to
[/var/lib/cassandra/data/disk1/kt_ks/new_table-44263b406bf111ed8bd9df80ace3677a/nb-5685-big,]
to level=0. 140.672GiB to 134.554GiB (~95% of original) in 13,894,920ms.
Read Throughput = 10.367MiB/s, Write Throughput = 9.916MiB/s, Row
Throughput = ~-80,450/s. 1,425,146,188 total partitions merged to
1,378,482,110. Partition merge counts were {1:1339149089, 2:32001964,
3:7331057, }
INFO [NonPeriodicTasks:1] 2023-04-04 04:39:08,151 SSTable.java:111 -
Deleting sstable:
/var/lib/cassandra/data/disk1/kt_ks/new_table-44263b406bf111ed8bd9df80ace3677a/nb-5546-big
INFO [NonPeriodicTasks:1] 2023-04-04 04:39:09,399 SSTable.java:111 -
Deleting sstable:
/var/lib/cassandra/data/disk1/kt_ks/new_table-44263b406bf111ed8bd9df80ace3677a/nb-5424-big
INFO [NonPeriodicTasks:1] 2023-04-04 04:39:10,563 SSTable.java:111 -
Deleting sstable:
/var/lib/cassandra/data/disk1/kt_ks/new_table-44263b406bf111ed8bd9df80ace3677a/nb-5677-big
INFO [NonPeriodicTasks:1] 2023-04-04 04:39:12,143 SSTable.java:111 -
Deleting sstable:
/var/lib/cassandra/data/disk1/kt_ks/new_table-44263b406bf111ed8bd9df80ace3677a/nb-3360-big
INFO [NonPeriodicTasks:1] 2023-04-04 04:39:12,754 SSTable.java:111 -
Deleting sstable:
/var/lib/cassandra/data/disk1/kt_ks/new_table-44263b406bf111ed8bd9df80ace3677a/nb-5604-big
INFO [NonPeriodicTasks:1] 2023-04-04 04:39:13,328 SSTable.java:111 -
Deleting sstable:
/var/lib/cassandra/data/disk1/kt_ks/new_table-44263b406bf111ed8bd9df80ace3677a/nb-5572-big
INFO [NonPeriodicTasks:1] 2023-04-04 04:39:13,340 SSTable.java:111 -
Deleting sstable:
/var/lib/cassandra/data/disk1/kt_ks/new_table-44263b406bf111ed8bd9df80ace3677a/nb-5684-big
INFO [NonPeriodicTasks:1] 2023-04-04 04:39:13,363 SSTable.java:111 -
Deleting sstable:
/var/lib/cassandra/data/disk1/kt_ks/new_table-44263b406bf111ed8bd9df80ace3677a/nb-5630-big
INFO [NonPeriodicTasks:1] 2023-04-04 04:39:14,492 SSTable.java:111 -
Deleting sstable:
/var/lib/cassandra/data/disk1/kt_ks/new_table-44263b406bf111ed8bd9df80ace3677a/nb-4991-big
INFO [NonPeriodicTasks:1] 2023-04-04 04:39:16,338 SSTable.java:111 -
Deleting sstable:
/var/lib/cassandra/data/disk1/kt_ks/new_table-44263b406bf111ed8bd9df80ace3677a/nb-2537-big
INFO [NonPeriodicTasks:1] 2023-04-04 04:39:18,287 SSTable.java:111 -
Deleting sstable:
/var/lib/cassandra/data/disk1/kt_ks/new_table-44263b406bf111ed8bd9df80ace3677a/nb-4171-big
INFO [NonPeriodicTasks:1] 2023-04-04 04:39:21,467 SSTable.java:111 -
Deleting sstable:
/var/lib/cassandra/data/disk1/kt_ks/new_table-44263b406bf111ed8bd9df80ace3677a/nb-1767-big
Am I missing something about how/when sstables are deleted? Is there
something I can do to control when that delete happens?
gil
