On Cassandra 2.2.11, I have a table that uses LeveledCompactionStrategy and that gets written to continuously. If I list the files in its data directory, I see something like this
-rw-r--r-- 1 acassy agroup 161733811 Jul 31 18:46 lb-135346-big-Data.db -rw-r--r-- 1 acassy agroup 159626222 Jul 31 02:53 lb-135297-big-Data.db -rw-r--r-- 1 acassy agroup 209892692 Jul 31 02:53 lb-135296-big-Data.db -rw-r--r-- 1 acassy agroup 209758372 Jul 31 02:53 lb-135295-big-Data.db -rw-r--r-- 1 acassy agroup 210109976 Jul 31 02:52 lb-135294-big-Data.db -rw-r--r-- 1 acassy agroup 209871524 Jul 31 02:52 lb-135293-big-Data.db -rw-r--r-- 1 acassy agroup 209889455 Jul 31 02:51 lb-135292-big-Data.db -rw-r--r-- 1 acassy agroup 209771979 Jul 31 02:51 lb-135291-big-Data.db -rw-r--r-- 1 acassy agroup 96253626 Jul 31 02:51 lb-135290-big-Data.db -rw-r--r-- 1 acassy agroup 209880076 Jul 31 02:51 lb-135289-big-Data.db -rw-r--r-- 1 acassy agroup 210007576 Jul 31 02:50 lb-135288-big-Data.db -rw-r--r-- 1 acassy agroup 209752707 Jul 31 02:50 lb-135287-big-Data.db -rw-r--r-- 1 acassy agroup 209832244 Jul 31 02:50 lb-135286-big-Data.db -rw-r--r-- 1 acassy agroup 209786246 Jul 31 02:50 lb-135285-big-Data.db -rw-r--r-- 1 acassy agroup 209786623 Jul 31 02:50 lb-135284-big-Data.db -rw-r--r-- 1 acassy agroup 209773999 Jul 31 02:49 lb-135282-big-Data.db -rw-r--r-- 1 acassy agroup 209769307 Jul 31 02:49 lb-135281-big-Data.db -rw-r--r-- 1 acassy agroup 209774780 Jul 31 02:49 lb-135280-big-Data.db -rw-r--r-- 1 acassy agroup 157218909 Jul 30 02:56 lb-135211-big-Data.db -rw-r--r-- 1 acassy agroup 210016992 Jul 30 02:56 lb-135210-big-Data.db -rw-r--r-- 1 acassy agroup 209723709 Jul 30 02:56 lb-135209-big-Data.db -rw-r--r-- 1 acassy agroup 209799709 Jul 30 02:56 lb-135208-big-Data.db -rw-r--r-- 1 acassy agroup 35399539 Jul 29 04:05 lb-135128-big-Data.db -rw-r--r-- 1 acassy agroup 209737007 Jul 29 04:05 lb-135127-big-Data.db -rw-r--r-- 1 acassy agroup 209741955 Jul 29 04:05 lb-135126-big-Data.db -rw-r--r-- 1 acassy agroup 209837894 Jul 29 04:04 lb-135125-big-Data.db -rw-r--r-- 1 acassy agroup 209775097 Jul 29 04:04 lb-135124-big-Data.db -rw-r--r-- 1 acassy agroup 2762187 Jul 28 09:48 lb-135050-big-Data.db -rw-r--r-- 1 acassy agroup 209736820 Jul 28 09:48 lb-135049-big-Data.db -rw-r--r-- 1 acassy agroup 209740897 Jul 28 09:47 lb-135048-big-Data.db -rw-r--r-- 1 acassy agroup 209765920 Jul 28 09:47 lb-135047-big-Data.db -rw-r--r-- 1 acassy agroup 210085882 Jul 28 09:46 lb-135046-big-Data.db -rw-r--r-- 1 acassy agroup 209901085 Jul 28 09:46 lb-135045-big-Data.db -rw-r--r-- 1 acassy agroup 209861149 Jul 28 09:46 lb-135044-big-Data.db -rw-r--r-- 1 acassy agroup 209723151 Jul 28 09:45 lb-135043-big-Data.db -rw-r--r-- 1 acassy agroup 209733066 Jul 28 09:45 lb-135042-big-Data.db -rw-r--r-- 1 acassy agroup 78406141 Jul 27 13:29 lb-134962-big-Data.db -rw-r--r-- 1 acassy agroup 209781961 Jul 27 13:29 lb-134961-big-Data.db -rw-r--r-- 1 acassy agroup 186161072 Jul 26 16:27 lb-134881-big-Data.db -rw-r--r-- 1 acassy agroup 209857207 Jul 26 16:27 lb-134880-big-Data.db -rw-r--r-- 1 acassy agroup 42059209 Jul 25 19:32 lb-134800-big-Data.db -rw-r--r-- 1 acassy agroup 210004006 Jul 25 19:32 lb-134799-big-Data.db -rw-r--r-- 1 acassy agroup 87893551 Jul 24 23:08 lb-134721-big-Data.db -rw-r--r-- 1 acassy agroup 209827743 Jul 24 23:08 lb-134720-big-Data.db -rw-r--r-- 1 acassy agroup 209734295 Jul 24 23:08 lb-134719-big-Data.db -rw-r--r-- 1 acassy agroup 209883247 Jul 24 23:07 lb-134718-big-Data.db -rw-r--r-- 1 acassy agroup 209738278 Jul 24 23:07 lb-134717-big-Data.db -rw-r--r-- 1 acassy agroup 158983134 Jul 21 10:25 lb-134404-big-Data.db -rw-r--r-- 1 acassy agroup 209740532 Jul 21 10:25 lb-134403-big-Data.db -rw-r--r-- 1 acassy agroup 209725876 Jul 21 10:25 lb-134402-big-Data.db -rw-r--r-- 1 acassy agroup 72250507 Jul 18 02:10 lb-134062-big-Data.db -rw-r--r-- 1 acassy agroup 209827278 Jul 17 09:13 lb-133986-big-Data.db -rw-r--r-- 1 acassy agroup 88312194 Jul 13 20:30 lb-133665-big-Data.db -rw-r--r-- 1 acassy agroup 89114556 Jul 13 01:12 lb-133589-big-Data.db -rw-r--r-- 1 acassy agroup 175385370 Jul 12 08:44 lb-133518-big-Data.db -rw-r--r-- 1 acassy agroup 209742099 Jul 11 14:02 lb-133441-big-Data.db -rw-r--r-- 1 acassy agroup 209733254 Jul 10 18:06 lb-133361-big-Data.db -rw-r--r-- 1 acassy agroup 209724386 Jul 10 18:06 lb-133360-big-Data.db -rw-r--r-- 1 acassy agroup 57820535 Nov 28 2016 lb-99553-big-Data.db -rw-r--r-- 1 acassy agroup 209734839 Nov 28 2016 lb-99552-big-Data.db -rw-r--r-- 1 acassy agroup 209857899 Nov 28 2016 lb-99551-big-Data.db -rw-r--r-- 1 acassy agroup 209751188 Nov 28 2016 lb-99550-big-Data.db -rw-r--r-- 1 acassy agroup 209731744 Nov 28 2016 lb-99549-big-Data.db -rw-r--r-- 1 acassy agroup 209736307 Nov 28 2016 lb-99548-big-Data.db -rw-r--r-- 1 acassy agroup 209799584 Nov 28 2016 lb-99547-big-Data.db -rw-r--r-- 1 acassy agroup 210050527 Nov 28 2016 lb-99546-big-Data.db -rw-r--r-- 1 acassy agroup 209731049 Nov 28 2016 lb-99545-big-Data.db -rw-r--r-- 1 acassy agroup 209792768 Nov 28 2016 lb-99544-big-Data.db -rw-r--r-- 1 acassy agroup 209739117 Nov 28 2016 lb-99543-big-Data.db -rw-r--r-- 1 acassy agroup 209753948 Nov 28 2016 lb-99542-big-Data.db -rw-r--r-- 1 acassy agroup 209772674 Nov 28 2016 lb-99541-big-Data.db -rw-r--r-- 1 acassy agroup 209793439 Nov 28 2016 lb-99540-big-Data.db -rw-r--r-- 1 acassy agroup 209719742 Nov 28 2016 lb-99539-big-Data.db -rw-r--r-- 1 acassy agroup 209784762 Nov 28 2016 lb-99538-big-Data.db -rw-r--r-- 1 acassy agroup 209744155 Nov 28 2016 lb-99536-big-Data.db -rw-r--r-- 1 acassy agroup 209878115 Nov 28 2016 lb-99535-big-Data.db -rw-r--r-- 1 acassy agroup 209749096 Nov 28 2016 lb-99534-big-Data.db -rw-r--r-- 1 acassy agroup 209859702 Nov 28 2016 lb-99533-big-Data.db -rw-r--r-- 1 acassy agroup 209834936 Nov 28 2016 lb-99532-big-Data.db -rw-r--r-- 1 acassy agroup 209768726 Nov 28 2016 lb-99531-big-Data.db -rw-r--r-- 1 acassy agroup 209754728 Nov 28 2016 lb-99530-big-Data.db -rw-r--r-- 1 acassy agroup 209718788 Nov 28 2016 lb-99529-big-Data.db -rw-r--r-- 1 acassy agroup 209769816 Nov 28 2016 lb-99528-big-Data.db -rw-r--r-- 1 acassy agroup 170810315 Nov 25 2016 lb-98259-big-Data.db -rw-r--r-- 1 acassy agroup 209749227 Nov 25 2016 lb-98258-big-Data.db -rw-r--r-- 1 acassy agroup 209735521 Nov 23 2016 lb-97557-big-Data.db -rw-r--r-- 1 acassy agroup 209748060 Nov 23 2016 lb-97550-big-Data.db -rw-r--r-- 1 acassy agroup 209724471 Nov 23 2016 lb-97445-big-Data.db -rw-r--r-- 1 acassy agroup 209869523 Nov 18 2016 lb-95132-big-Data.db -rw-r--r-- 1 acassy agroup 209809636 Nov 17 2016 lb-94927-big-Data.db -rw-r--r-- 1 acassy agroup 209758214 Nov 17 2016 lb-94873-big-Data.db -rw-r--r-- 1 acassy agroup 209804833 Nov 16 2016 lb-94452-big-Data.db -rw-r--r-- 1 acassy agroup 110494743 Oct 1 2016 lb-75515-big-Data.db -rw-r--r-- 1 acassy agroup 209777568 Oct 1 2016 lb-75514-big-Data.db -rw-r--r-- 1 acassy agroup 209690277 Aug 16 2016 lb-50882-big-Data.db -rw-r--r-- 1 acassy agroup 209743321 Aug 16 2016 lb-50879-big-Data.db -rw-r--r-- 1 acassy agroup 209708311 Aug 16 2016 lb-50877-big-Data.db -rw-r--r-- 1 acassy agroup 200324269 Aug 16 2016 lb-50614-big-Data.db -rw-r--r-- 1 acassy agroup 201124079 Aug 16 2016 lb-50613-big-Data.db -rw-r--r-- 1 acassy agroup 201165532 Aug 16 2016 lb-50612-big-Data.db -rw-r--r-- 1 acassy agroup 201079038 Aug 16 2016 lb-50611-big-Data.db -rw-r--r-- 1 acassy agroup 201189531 Aug 16 2016 lb-50610-big-Data.db -rw-r--r-- 1 acassy agroup 201091465 Aug 16 2016 lb-50609-big-Data.db -rw-r--r-- 1 acassy agroup 201147689 Aug 16 2016 lb-50607-big-Data.db -rw-r--r-- 1 acassy agroup 201072987 Aug 16 2016 lb-50606-big-Data.db -rw-r--r-- 1 acassy agroup 201234706 Aug 16 2016 lb-50604-big-Data.db -rw-r--r-- 1 acassy agroup 201118109 Aug 16 2016 lb-50603-big-Data.db Notice all those files from 2016. These never get compacted away. If I output nodetool cfstats, I see there are currently four SSTable levels. Table: MyTableSSTable count: 70SSTables in each level: [1, 10, 47, 50, 0, 0, 0, 0, 0] (200MB each) sstablemetadata tells me those August files are in the 3rd level. I can also tell through lsof that Cassandra has an open handle to all of these files. Why isn't Cassandra including these files in its compactions? Has the strategy simply not reached its threshold for the next level? That seems very unlikely in almost a year. This particular table has TTL for all its rows, but I've seen this behaviour in other tables that don't. If I issue a compaction with nodetool compact, they disappear. What's going on?
