Writes slower then reads

2012-01-05 Thread R. Verlangen
Hi there, I'm running a cassandra 0.8.6 cluster with 2 nodes (in 2 DC's), RF = 2. Actual data on the nodes is only 1GB. Disk latency 1ms. Disk throughput ~ 0.4MB/s. OS load always below 1 (on a 8 core machine with 16GB ram). When I'm running my writes against the cluster with cl = ONE all reads

Re: Writes slower then reads

2012-01-05 Thread Philippe
What can you see in vmstat/dstat ? Le 5 janv. 2012 11:58, R. Verlangen ro...@us2.nl a écrit : Hi there, I'm running a cassandra 0.8.6 cluster with 2 nodes (in 2 DC's), RF = 2. Actual data on the nodes is only 1GB. Disk latency 1ms. Disk throughput ~ 0.4MB/s. OS load always below 1 (on a 8

Re: Writes slower then reads

2012-01-05 Thread R. Verlangen
As I posted this I noticed that the other node's CPU is running high on some other cronjobs (every couple of minutes to 60% usage). Is the lack of more CPU cycles a problem in this case? Robin 2012/1/5 R. Verlangen ro...@us2.nl CPU is idle ( 10% usage). Disk reads occasionally blocks over

Re: Writes slower then reads

2012-01-05 Thread Philippe
Depending on the CL you're reading at it will yes : if the CL requires that the slow node create a digest of the data and send it to the coordinator then it might explain the poor performance on reads. What is your read CL ? 2012/1/5 R. Verlangen ro...@us2.nl As I posted this I noticed that the

Re: Writes slower then reads

2012-01-05 Thread R. Verlangen
I'm also reading with CL = ONE 2012/1/5 Philippe watche...@gmail.com Depending on the CL you're reading at it will yes : if the CL requires that the slow node create a digest of the data and send it to the coordinator then it might explain the poor performance on reads. What is your read CL

Re: Writes slower then reads

2012-01-05 Thread Philippe
What if you shutdown the cassandra service on the slow node, does that improve your read performance ? If it does then that sole node is responsible for the slow down because it can't act as a coordinator fast enough. 2012/1/5 R. Verlangen ro...@us2.nl I'm also reading with CL = ONE 2012/1/5

Re: Writes slower then reads

2012-01-05 Thread R. Verlangen
It does not appear to affect the response time, certainly not in a positive way. 2012/1/5 Philippe watche...@gmail.com What if you shutdown the cassandra service on the slow node, does that improve your read performance ? If it does then that sole node is responsible for the slow down because

Re: Writes slower then reads

2012-01-05 Thread Philippe
You may be overloading the cluster though... My hypothesis is that your traffic is being spread across your node and that one slow node is slowing down the fraction of traffic that goes to that node (when it's acting as coordinator). So what I would do is reduce the read load a lot to make sure I

Re: Writes slower then reads

2012-01-05 Thread R. Verlangen
The write and read load is very minimal the moment. Roughly 10 writes + 10 reads / second. So 20 operations per second. Don't think that overloads my cluster, does it? 2012/1/5 Philippe watche...@gmail.com You may be overloading the cluster though... My hypothesis is that your traffic is

Re: Writes slower then reads

2012-01-05 Thread Philippe
Unless you are doing huge batches no... don't have any other idea for now... 2012/1/5 R. Verlangen ro...@us2.nl The write and read load is very minimal the moment. Roughly 10 writes + 10 reads / second. So 20 operations per second. Don't think that overloads my cluster, does it? 2012/1/5

Re: Writes slower then reads

2012-01-05 Thread aaron morton
What happens when you turn off the cron jobs ? Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 6/01/2012, at 6:57 AM, Philippe wrote: Unless you are doing huge batches no... don't have any other idea for now... 2012/1/5 R. Verlangen