Anuj, Jeff, thank you both, Although harmless, sounds like it's time for an upgrade. The ticket suggests that 2.0.17 is not affected.
Thank you guys! On Fri, Dec 11, 2015 at 5:25 PM, Jeff Jirsa <jeff.ji...@crowdstrike.com> wrote: > Same bug also affects 2.0.16 - > https://issues.apache.org/jira/browse/CASSANDRA-9662 > > From: Jeff Jirsa > Reply-To: <user@cassandra.apache.org> > Date: Friday, December 11, 2015 at 9:12 AM > To: "user@cassandra.apache.org" > Subject: Re: Thousands of pending compactions using STCS > > There were a few buggy versions in 2.1 (2.1.7, 2.1.8, I believe) that > showed this behavior. The number of pending compactions was artificially > high, and not meaningful. As long as they number of –Data.db sstables > remains normal, compaction is keeping up and you’re fine. > > - Jeff > > From: Vasileios Vlachos > Reply-To: "user@cassandra.apache.org" > Date: Friday, December 11, 2015 at 8:28 AM > To: "user@cassandra.apache.org" > Subject: Thousands of pending compactions using STCS > > Hello, > > We use Nagios and MX4J for the majority of the monitoring we do for > Cassandra (version: 2.0.16). For compactions we hit the following URL: > > > http://${cassandra_host}:8081/mbean?objectname=org.apache.cassandra.db%3Atype%3DCompactionManager > > and check the PendingTasks counter's value. > > We have noticed that occasionally one or more nodes will report back that > they have thousands of pending compactions. We have 11 KS in the cluster > and a total of 109 *Data.db files under /var/lib/cassandra/data which > gives approximately 10 SSTables per KS. That makes us think that having > thousands of pending compactions seems unrealistic given the number of > SSTables we seem to have at any given time in each KS/CF directory. The > logs show a lot of flush and compaction activity but we don't think that's > unusual. Also, each CF is configured to have min_compaction_threshold = 2 > and max_compaction_threshold = 32. The two screenshots below show a > cluster-wide view of pending compactions. Attached you can find the XML > files which contain the data from the MX4J console. > > [image: Inline image 2] > > And this is from the same graph, but I've selected the time period after > 14:00 in order to show what the real compaction activity looks like when > not skewed by the incredibly high number of pending compactions as shown > above: > [image: Inline image 3] > > Has anyone else experienced something similar? Is there something else we > can do to see if this is something wrong with our cluster? > > Thanks in advance for any help! > > Vasilis >
