I'm looking into creation of monitoring thresholds for cassandra to report on its health. Does it make sense to set an alert threshold on compaction stats? If so, would setting it to a value equal to or greater than concurrent compactions make sense?
Thanks, Greg On Mon, Jun 9, 2014 at 2:14 PM, S C <as...@outlook.com> wrote: > Thank you all for quick responses. > ------------------------------ > From: clohf...@blackbirdit.com > Subject: Re: high pending compactions > Date: Mon, 9 Jun 2014 14:11:36 -0500 > To: user@cassandra.apache.org > > Bean: org.apache.cassandra.db.CompactionManager > > also nodetool compactionstats gives you how many are in the queue + > estimate of how many will be needed. > > in 1.1 you will OOM *far* before you hit the limit,. In theory though, > the compaction executor is a little special cased and will actually throw > an exception (normally it will block) > > Chris > > On Jun 9, 2014, at 7:49 AM, S C <as...@outlook.com> wrote: > > Thank you all for valuable suggestions. Couple more questions, > > How to check the compaction queue? MBean/C* system log ? > What happens if the queue is full? > > ------------------------------ > From: colinkuo...@gmail.com > Date: Mon, 9 Jun 2014 18:53:41 +0800 > Subject: Re: high pending compactions > To: user@cassandra.apache.org > > As Jake suggested, you could firstly increase > "compaction_throughput_mb_per_sec" and "concurrent_compactions" to suitable > values if system resource is allowed. From my understanding, major > compaction will internally acquire lock before running compaction. In your > case, there might be a major compaction blocking the pending following > compaction tasks. You could check the result of "nodetool compactionstats" > and C* system log for double confirm. > > If the running compaction is compacting wide row for a long time, you > could try to tune "in_memory_compaction_limit_in_mb" value. > > Thanks, > > > > On Sun, Jun 8, 2014 at 11:27 PM, S C <as...@outlook.com> wrote: > > I am using Cassandra 1.1 (sorry bit old) and I am seeing high pending > compaction count. "pending tasks: 67" while active compaction tasks are > not more than 5. I have a 24CPU machine. Shouldn't I be seeing more > compactions? Is this a pattern of high writes and compactions backing up? > How can I improve this? Here are my thoughts. > > > 1. Increase memtable_total_space_in_mb > 2. Increase compaction_throughput_mb_per_sec > 3. Increase concurrent_compactions > > > Sorry if this was discussed already. Any pointers is much appreciated. > > Thanks, > Kumar > > >
