Re: Would warnings about overlapping SStables explain high pending compactions?
Not really What version are you on? Do you have pending compactions and no ongoing compactions? /Marcus On Wed, Sep 24, 2014 at 11:35 PM, Donald Smith donald.sm...@audiencescience.com wrote: On one of our nodes we have lots of pending compactions (499).In the past we’ve seen pending compactions go up to 2400 and all the way back down again. Investigating, I saw warnings such as the following in the logs about overlapping SStables and about needing to run “nodetool scrub” on a table. Would the overlapping SStables explain the pending compactions? WARN [RMI TCP Connection(2)-10.5.50.30] 2014-09-24 09:14:11,207 LeveledManifest.java (line 154) At level 1, SSTableReader(path='/data/data/XYZ/ABC/XYZ-ABC-jb-388233-Data.db') [DecoratedKey(-6112875836465333229, 3366636664393031646263356234663832383264616561666430383739383738), DecoratedKey(-4509284829153070912, 3366336562386339376664376633353635333432636662373739626465393636)] overlaps SSTableReader(path='/data/data/XYZ/ABC/XYZ-ABC_blob-jb-388150-Data.db') [DecoratedKey(-4834684725563291584, 336633623334363664363632666365303664333936336337343566373838), DecoratedKey(-4136919579566299218, 3366613535646662343235336335633862666530316164323232643765323934)]. This could be caused by a bug in Cassandra 1.1.0 .. 1.1.3 or due to the fact that you have dropped sstables from another node into the data directory. Sending back to L0. If you didn't drop in sstables, and have not yet run scrub, you should do so since you may also have rows out-of-order within an sstable Thanks *Donald A. Smith* | Senior Software Engineer P: 425.201.3900 x 3866 C: (206) 819-5965 F: (646) 443-2333 dona...@audiencescience.com [image: AudienceScience]
RE: Would warnings about overlapping SStables explain high pending compactions?
Version 2.0.9. We have 11 ongoing compactions on that node. From: Marcus Eriksson [mailto:krum...@gmail.com] Sent: Thursday, September 25, 2014 12:45 AM To: user@cassandra.apache.org Subject: Re: Would warnings about overlapping SStables explain high pending compactions? Not really What version are you on? Do you have pending compactions and no ongoing compactions? /Marcus On Wed, Sep 24, 2014 at 11:35 PM, Donald Smith donald.sm...@audiencescience.commailto:donald.sm...@audiencescience.com wrote: On one of our nodes we have lots of pending compactions (499).In the past we’ve seen pending compactions go up to 2400 and all the way back down again. Investigating, I saw warnings such as the following in the logs about overlapping SStables and about needing to run “nodetool scrub” on a table. Would the overlapping SStables explain the pending compactions? WARN [RMI TCP Connection(2)-10.5.50.30] 2014-09-24 09:14:11,207 LeveledManifest.java (line 154) At level 1, SSTableReader(path='/data/data/XYZ/ABC/XYZ-ABC-jb-388233-Data.db') [DecoratedKey(-6112875836465333229, 3366636664393031646263356234663832383264616561666430383739383738), DecoratedKey(-4509284829153070912, 3366336562386339376664376633353635333432636662373739626465393636)] overlaps SSTableReader(path='/data/data/XYZ/ABC/XYZ-ABC_blob-jb-388150-Data.db') [DecoratedKey(-4834684725563291584, 336633623334363664363632666365303664333936336337343566373838), DecoratedKey(-4136919579566299218, 3366613535646662343235336335633862666530316164323232643765323934)]. This could be caused by a bug in Cassandra 1.1.0 .. 1.1.3 or due to the fact that you have dropped sstables from another node into the data directory. Sending back to L0. If you didn't drop in sstables, and have not yet run scrub, you should do so since you may also have rows out-of-order within an sstable Thanks Donald A. Smith | Senior Software Engineer P: 425.201.3900 x 3866tel:425.201.3900%20x%203866 C: (206) 819-5965tel:%28206%29%20819-5965 F: (646) 443-2333tel:%28646%29%20443-2333 dona...@audiencescience.commailto:dona...@audiencescience.com [AudienceScience]
Would warnings about overlapping SStables explain high pending compactions?
On one of our nodes we have lots of pending compactions (499).In the past we've seen pending compactions go up to 2400 and all the way back down again. Investigating, I saw warnings such as the following in the logs about overlapping SStables and about needing to run nodetool scrub on a table. Would the overlapping SStables explain the pending compactions? WARN [RMI TCP Connection(2)-10.5.50.30] 2014-09-24 09:14:11,207 LeveledManifest.java (line 154) At level 1, SSTableReader(path='/data/data/XYZ/ABC/XYZ-ABC-jb-388233-Data.db') [DecoratedKey(-6112875836465333229, 3366636664393031646263356234663832383264616561666430383739383738), DecoratedKey(-4509284829153070912, 3366336562386339376664376633353635333432636662373739626465393636)] overlaps SSTableReader(path='/data/data/XYZ/ABC/XYZ-ABC_blob-jb-388150-Data.db') [DecoratedKey(-4834684725563291584, 336633623334363664363632666365303664333936336337343566373838), DecoratedKey(-4136919579566299218, 3366613535646662343235336335633862666530316164323232643765323934)]. This could be caused by a bug in Cassandra 1.1.0 .. 1.1.3 or due to the fact that you have dropped sstables from another node into the data directory. Sending back to L0. If you didn't drop in sstables, and have not yet run scrub, you should do so since you may also have rows out-of-order within an sstable Thanks Donald A. Smith | Senior Software Engineer P: 425.201.3900 x 3866 C: (206) 819-5965 F: (646) 443-2333 dona...@audiencescience.commailto:dona...@audiencescience.com [AudienceScience]
Re: high pending compactions
Number of compactions can spike for perfectly reasonable scenarios (ie after a repair or bootstrap) so no matter what you set it to there will likely be periodic false alarms. I would however put a number 15 on the limit of any thread pool backlog as matter of principle until woken up too many times in middle of night. concurrent compactors will likely be to low (depending on number of cores). --- Chris Lohfink On Jul 14, 2014, at 7:31 PM, Greg Bone gbon...@gmail.com wrote: I'm looking into creation of monitoring thresholds for cassandra to report on its health. Does it make sense to set an alert threshold on compaction stats? If so, would setting it to a value equal to or greater than concurrent compactions make sense? Thanks, Greg On Mon, Jun 9, 2014 at 2:14 PM, S C as...@outlook.com wrote: Thank you all for quick responses. From: clohf...@blackbirdit.com Subject: Re: high pending compactions Date: Mon, 9 Jun 2014 14:11:36 -0500 To: user@cassandra.apache.org Bean: org.apache.cassandra.db.CompactionManager also nodetool compactionstats gives you how many are in the queue + estimate of how many will be needed. in 1.1 you will OOM far before you hit the limit,. In theory though, the compaction executor is a little special cased and will actually throw an exception (normally it will block) Chris On Jun 9, 2014, at 7:49 AM, S C as...@outlook.com wrote: Thank you all for valuable suggestions. Couple more questions, How to check the compaction queue? MBean/C* system log ? What happens if the queue is full? From: colinkuo...@gmail.com Date: Mon, 9 Jun 2014 18:53:41 +0800 Subject: Re: high pending compactions To: user@cassandra.apache.org As Jake suggested, you could firstly increase compaction_throughput_mb_per_sec and concurrent_compactions to suitable values if system resource is allowed. From my understanding, major compaction will internally acquire lock before running compaction. In your case, there might be a major compaction blocking the pending following compaction tasks. You could check the result of nodetool compactionstats and C* system log for double confirm. If the running compaction is compacting wide row for a long time, you could try to tune in_memory_compaction_limit_in_mb value. Thanks, On Sun, Jun 8, 2014 at 11:27 PM, S C as...@outlook.com wrote: I am using Cassandra 1.1 (sorry bit old) and I am seeing high pending compaction count. pending tasks: 67 while active compaction tasks are not more than 5. I have a 24CPU machine. Shouldn't I be seeing more compactions? Is this a pattern of high writes and compactions backing up? How can I improve this? Here are my thoughts. Increase memtable_total_space_in_mb Increase compaction_throughput_mb_per_sec Increase concurrent_compactions Sorry if this was discussed already. Any pointers is much appreciated. Thanks, Kumar
Re: high pending compactions
I'm looking into creation of monitoring thresholds for cassandra to report on its health. Does it make sense to set an alert threshold on compaction stats? If so, would setting it to a value equal to or greater than concurrent compactions make sense? Thanks, Greg On Mon, Jun 9, 2014 at 2:14 PM, S C as...@outlook.com wrote: Thank you all for quick responses. -- From: clohf...@blackbirdit.com Subject: Re: high pending compactions Date: Mon, 9 Jun 2014 14:11:36 -0500 To: user@cassandra.apache.org Bean: org.apache.cassandra.db.CompactionManager also nodetool compactionstats gives you how many are in the queue + estimate of how many will be needed. in 1.1 you will OOM *far* before you hit the limit,. In theory though, the compaction executor is a little special cased and will actually throw an exception (normally it will block) Chris On Jun 9, 2014, at 7:49 AM, S C as...@outlook.com wrote: Thank you all for valuable suggestions. Couple more questions, How to check the compaction queue? MBean/C* system log ? What happens if the queue is full? -- From: colinkuo...@gmail.com Date: Mon, 9 Jun 2014 18:53:41 +0800 Subject: Re: high pending compactions To: user@cassandra.apache.org As Jake suggested, you could firstly increase compaction_throughput_mb_per_sec and concurrent_compactions to suitable values if system resource is allowed. From my understanding, major compaction will internally acquire lock before running compaction. In your case, there might be a major compaction blocking the pending following compaction tasks. You could check the result of nodetool compactionstats and C* system log for double confirm. If the running compaction is compacting wide row for a long time, you could try to tune in_memory_compaction_limit_in_mb value. Thanks, On Sun, Jun 8, 2014 at 11:27 PM, S C as...@outlook.com wrote: I am using Cassandra 1.1 (sorry bit old) and I am seeing high pending compaction count. pending tasks: 67 while active compaction tasks are not more than 5. I have a 24CPU machine. Shouldn't I be seeing more compactions? Is this a pattern of high writes and compactions backing up? How can I improve this? Here are my thoughts. 1. Increase memtable_total_space_in_mb 2. Increase compaction_throughput_mb_per_sec 3. Increase concurrent_compactions Sorry if this was discussed already. Any pointers is much appreciated. Thanks, Kumar
Re: high pending compactions
As Jake suggested, you could firstly increase compaction_throughput_mb_per_sec and concurrent_compactions to suitable values if system resource is allowed. From my understanding, major compaction will internally acquire lock before running compaction. In your case, there might be a major compaction blocking the pending following compaction tasks. You could check the result of nodetool compactionstats and C* system log for double confirm. If the running compaction is compacting wide row for a long time, you could try to tune in_memory_compaction_limit_in_mb value. Thanks, On Sun, Jun 8, 2014 at 11:27 PM, S C as...@outlook.com wrote: I am using Cassandra 1.1 (sorry bit old) and I am seeing high pending compaction count. pending tasks: 67 while active compaction tasks are not more than 5. I have a 24CPU machine. Shouldn't I be seeing more compactions? Is this a pattern of high writes and compactions backing up? How can I improve this? Here are my thoughts. 1. Increase memtable_total_space_in_mb 2. Increase compaction_throughput_mb_per_sec 3. Increase concurrent_compactions Sorry if this was discussed already. Any pointers is much appreciated. Thanks, Kumar
RE: high pending compactions
Thank you all for valuable suggestions. Couple more questions, How to check the compaction queue? MBean/C* system log ?What happens if the queue is full? From: colinkuo...@gmail.com Date: Mon, 9 Jun 2014 18:53:41 +0800 Subject: Re: high pending compactions To: user@cassandra.apache.org As Jake suggested, you could firstly increase compaction_throughput_mb_per_sec and concurrent_compactions to suitable values if system resource is allowed. From my understanding, major compaction will internally acquire lock before running compaction. In your case, there might be a major compaction blocking the pending following compaction tasks. You could check the result of nodetool compactionstats and C* system log for double confirm. If the running compaction is compacting wide row for a long time, you could try to tune in_memory_compaction_limit_in_mb value. Thanks, On Sun, Jun 8, 2014 at 11:27 PM, S C as...@outlook.com wrote: I am using Cassandra 1.1 (sorry bit old) and I am seeing high pending compaction count. pending tasks: 67 while active compaction tasks are not more than 5. I have a 24CPU machine. Shouldn't I be seeing more compactions? Is this a pattern of high writes and compactions backing up? How can I improve this? Here are my thoughts. Increase memtable_total_space_in_mbIncrease compaction_throughput_mb_per_secIncrease concurrent_compactions Sorry if this was discussed already. Any pointers is much appreciated. Thanks,Kumar
Re: high pending compactions
Bean: org.apache.cassandra.db.CompactionManager also nodetool compactionstats gives you how many are in the queue + estimate of how many will be needed. in 1.1 you will OOM far before you hit the limit,. In theory though, the compaction executor is a little special cased and will actually throw an exception (normally it will block) Chris On Jun 9, 2014, at 7:49 AM, S C as...@outlook.com wrote: Thank you all for valuable suggestions. Couple more questions, How to check the compaction queue? MBean/C* system log ? What happens if the queue is full? From: colinkuo...@gmail.com Date: Mon, 9 Jun 2014 18:53:41 +0800 Subject: Re: high pending compactions To: user@cassandra.apache.org As Jake suggested, you could firstly increase compaction_throughput_mb_per_sec and concurrent_compactions to suitable values if system resource is allowed. From my understanding, major compaction will internally acquire lock before running compaction. In your case, there might be a major compaction blocking the pending following compaction tasks. You could check the result of nodetool compactionstats and C* system log for double confirm. If the running compaction is compacting wide row for a long time, you could try to tune in_memory_compaction_limit_in_mb value. Thanks, On Sun, Jun 8, 2014 at 11:27 PM, S C as...@outlook.com wrote: I am using Cassandra 1.1 (sorry bit old) and I am seeing high pending compaction count. pending tasks: 67 while active compaction tasks are not more than 5. I have a 24CPU machine. Shouldn't I be seeing more compactions? Is this a pattern of high writes and compactions backing up? How can I improve this? Here are my thoughts. Increase memtable_total_space_in_mb Increase compaction_throughput_mb_per_sec Increase concurrent_compactions Sorry if this was discussed already. Any pointers is much appreciated. Thanks, Kumar
RE: high pending compactions
Thank you all for quick responses. From: clohf...@blackbirdit.com Subject: Re: high pending compactions Date: Mon, 9 Jun 2014 14:11:36 -0500 To: user@cassandra.apache.org Bean: org.apache.cassandra.db.CompactionManager also nodetool compactionstats gives you how many are in the queue + estimate of how many will be needed. in 1.1 you will OOM far before you hit the limit,. In theory though, the compaction executor is a little special cased and will actually throw an exception (normally it will block) Chris On Jun 9, 2014, at 7:49 AM, S C as...@outlook.com wrote:Thank you all for valuable suggestions. Couple more questions, How to check the compaction queue? MBean/C* system log ?What happens if the queue is full? From: colinkuo...@gmail.com Date: Mon, 9 Jun 2014 18:53:41 +0800 Subject: Re: high pending compactions To: user@cassandra.apache.org As Jake suggested, you could firstly increase compaction_throughput_mb_per_sec and concurrent_compactions to suitable values if system resource is allowed. From my understanding, major compaction will internally acquire lock before running compaction. In your case, there might be a major compaction blocking the pending following compaction tasks. You could check the result of nodetool compactionstats and C* system log for double confirm. If the running compaction is compacting wide row for a long time, you could try to tune in_memory_compaction_limit_in_mb value. Thanks, On Sun, Jun 8, 2014 at 11:27 PM, S C as...@outlook.com wrote: I am using Cassandra 1.1 (sorry bit old) and I am seeing high pending compaction count. pending tasks: 67 while active compaction tasks are not more than 5. I have a 24CPU machine. Shouldn't I be seeing more compactions? Is this a pattern of high writes and compactions backing up? How can I improve this? Here are my thoughts. Increase memtable_total_space_in_mbIncrease compaction_throughput_mb_per_secIncrease concurrent_compactions Sorry if this was discussed already. Any pointers is much appreciated. Thanks,Kumar
high pending compactions
I am using Cassandra 1.1 (sorry bit old) and I am seeing high pending compaction count. pending tasks: 67 while active compaction tasks are not more than 5. I have a 24CPU machine. Shouldn't I be seeing more compactions? Is this a pattern of high writes and compactions backing up? How can I improve this? Here are my thoughts. Increase memtable_total_space_in_mbIncrease compaction_throughput_mb_per_secIncrease concurrent_compactions Sorry if this was discussed already. Any pointers is much appreciated. Thanks,Kumar
Re: high pending compactions
23 On Sunday, June 8, 2014, S C as...@outlook.com wrote: I am using Cassandra 1.1 (sorry bit old) and I am seeing high pending compaction count. pending tasks: 67 while active compaction tasks are not more than 5. I have a 24CPU machine. Shouldn't I be seeing more compactions? Is this a pattern of high writes and compactions backing up? How can I improve this? Here are my thoughts. 1. Increase memtable_total_space_in_mb 2. Increase compaction_throughput_mb_per_sec 3. Increase concurrent_compactions Sorry if this was discussed already. Any pointers is much appreciated. Thanks, Kumar -- http://twitter.com/tjake
Re: high pending compactions
Have you verified that these aren't stuck compactions? e.g. even under no load, they don't go away? -Bill On 06/08/2014 12:32 PM, Jake Luciani wrote: 23 On Sunday, June 8, 2014, S C as...@outlook.com mailto:as...@outlook.com wrote: I am using Cassandra 1.1 (sorry bit old) and I am seeing high pending compaction count. pending tasks: 67 while active compaction tasks are not more than 5. I have a 24CPU machine. Shouldn't I be seeing more compactions? Is this a pattern of high writes and compactions backing up? How can I improve this? Here are my thoughts. 1. Increase memtable_total_space_in_mb 2. Increase compaction_throughput_mb_per_sec 3. Increase concurrent_compactions Sorry if this was discussed already. Any pointers is much appreciated. Thanks, Kumar -- http://twitter.com/tjake
RE: high pending compactions
How to check if there are any stuck compactions? Date: Sun, 8 Jun 2014 12:43:45 -0400 From: wkat...@cs.rutgers.edu To: user@cassandra.apache.org Subject: Re: high pending compactions Have you verified that these aren't stuck compactions? e.g. even under no load, they don't go away? -Bill On 06/08/2014 12:32 PM, Jake Luciani wrote: 23 On Sunday, June 8, 2014, S C as...@outlook.com mailto:as...@outlook.com wrote: I am using Cassandra 1.1 (sorry bit old) and I am seeing high pending compaction count. pending tasks: 67 while active compaction tasks are not more than 5. I have a 24CPU machine. Shouldn't I be seeing more compactions? Is this a pattern of high writes and compactions backing up? How can I improve this? Here are my thoughts. 1. Increase memtable_total_space_in_mb 2. Increase compaction_throughput_mb_per_sec 3. Increase concurrent_compactions Sorry if this was discussed already. Any pointers is much appreciated. Thanks, Kumar -- http://twitter.com/tjake
