Re: Avoid Simultaneous Minor Compactions?

2011-08-23 Thread aaron morton
Change one thing at a time and work out what metric it is you want to improve. I would start with reducing compaction_throughput_mb_per_sec. Have a look in your logs for the Enqueuing flush of Memtable… messages, count up how many serialised bytes you are flushing and then check it against the

Re: Avoid Simultaneous Minor Compactions?

2011-08-22 Thread Jonathan Ellis
Specifically, look at compaction_throughput_mb_per_sec in cassandra.yaml On Mon, Aug 22, 2011 at 12:39 AM, Ryan King r...@twitter.com wrote: You should throttle your compactions to a sustainable level. -ryan On Sun, Aug 21, 2011 at 10:22 PM, Hefeng Yuan hfy...@rhapsody.com wrote: We just

Re: Avoid Simultaneous Minor Compactions?

2011-08-22 Thread Hefeng Yuan
Shall I lower this or increase it? Or probably ask in this way, do we suggest to let it run longer while using less CPU, or should we let it finish faster with more CPU usage? The problem we're facing is, with the default setting, they run slow and also eat a lot of CPU in the meanwhile. I'm

Avoid Simultaneous Minor Compactions?

2011-08-21 Thread Hefeng Yuan
We just noticed that at one time, 4 nodes were doing minor compaction together, each of them took 20~60 minutes. We're on 0.8.1, 6 nodes, RF5. This simultaneous compactions slowed down the whole cluster, we have local_quorum consistency level, therefore, dynamic_snitch is not helping us. Aside

Re: Avoid Simultaneous Minor Compactions?

2011-08-21 Thread Ryan King
You should throttle your compactions to a sustainable level. -ryan On Sun, Aug 21, 2011 at 10:22 PM, Hefeng Yuan hfy...@rhapsody.com wrote: We just noticed that at one time, 4 nodes were doing minor compaction together, each of them took 20~60 minutes. We're on 0.8.1, 6 nodes, RF5. This