I have one node in my cluster 2.2.7 (just upgraded from 2.2.6 hoping to fix issue) which seems to be stuck in a weird state -- with a large number of pending compactions and sstables. The node is compacting about 500gb/day, number of pending compactions is going up at about 50/day. It is at about 2300 pending compactions now. I have tried increasing number of compaction threads and the compaction throughput, which doesn't seem to help eliminate the many pending compactions.
I have tried running 'nodetool cleanup' and 'nodetool compact'. The latter has fixed the issue in the past, but most recently I was getting OOM errors, probably due to the large number of sstables. I upgraded to 2.2.7 and am no longer getting OOM errors, but also it does not resolve the issue. I do see this message in the logs: INFO [RMI TCP Connection(611)-10.9.2.218] 2016-08-17 01:50:01,985 > CompactionManager.java:610 - Cannot perform a full major compaction as > repaired and unrepaired sstables cannot be compacted together. These two > set of sstables will be compacted separately. > Below are the 'nodetool tablestats' comparing a normal and the problematic node. You can see problematic node has many many more sstables, and they are all in level 1. What is the best way to fix this? Can I just delete those sstables somehow then run a repair? > Normal node keyspace: mykeyspace > > Read Count: 0 > > Read Latency: NaN ms. > > Write Count: 31905656 > > Write Latency: 0.051713177939359714 ms. > > Pending Flushes: 0 > > Table: mytable > > SSTable count: 1908 > > SSTables in each level: [11/4, 20/10, 213/100, 1356/1000, 306, 0, >> 0, 0, 0] > > Space used (live): 301894591442 > > Space used (total): 301894591442 > > >> >> Problematic node > > Keyspace: mykeyspace > > Read Count: 0 > > Read Latency: NaN ms. > > Write Count: 30520190 > > Write Latency: 0.05171286705620116 ms. > > Pending Flushes: 0 > > Table: mytable > > SSTable count: 14105 > > SSTables in each level: [13039/4, 21/10, 206/100, 831, 0, 0, 0, 0, >> 0] > > Space used (live): 561143255289 > > Space used (total): 561143255289 > > Thanks, Ezra
