What happens when you turn off the cron jobs ? Cheers
----------------- Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 6/01/2012, at 6:57 AM, Philippe wrote: > Unless you are doing huge batches no... don't have any other idea for now... > > 2012/1/5 R. Verlangen <[email protected]> > The write and read load is very minimal the moment. Roughly 10 writes + 10 > reads / second. So 20 operations per second. Don't think that overloads my > cluster, does it? > > > 2012/1/5 Philippe <[email protected]> > You may be overloading the cluster though... > > My hypothesis is that your traffic is being spread across your node and that > one slow node is slowing down the fraction of traffic that goes to that node > (when it's acting as coordinator). > So what I would do is reduce the read load a lot to make sure I don't > overload the cluster and measure if I see a 1/RF improvement in response time > which would validate my hypothesis. > > > 2012/1/5 R. Verlangen <[email protected]> > > It does not appear to affect the response time, certainly not in a positive > way. > > > 2012/1/5 Philippe <[email protected]> > What if you shutdown the cassandra service on the slow node, does that > improve your read performance ? > If it does then that sole node is responsible for the slow down because it > can't act as a coordinator fast enough. > > 2012/1/5 R. Verlangen <[email protected]> > > I'm also reading with CL = ONE > > > 2012/1/5 Philippe <[email protected]> > Depending on the CL you're reading at it will yes : if the CL requires that > the "slow" node create a digest of the data and send it to the coordinator > then it might explain the poor performance on reads. What is your read CL ? > > 2012/1/5 R. Verlangen <[email protected]> > > As I posted this I noticed that the other node's CPU is running high on some > other cronjobs (every couple of minutes to 60% usage). Is the lack of more > CPU cycles a problem in this case? > > Robin > > 2012/1/5 R. Verlangen <[email protected]> > > CPU is idle (< 10% usage). Disk reads occasionally blocks over 32/64K. Writes > around 0-5MB per second. Network traffic 0.1 / 0.1 MB/s (in / out). Paging 0. > System int ~ 1300, csw ~ 2500. > > > 2012/1/5 Philippe <[email protected]> > What can you see in vmstat/dstat ? > > Le 5 janv. 2012 11:58, "R. Verlangen" <[email protected]> a écrit : > > Hi there, > > I'm running a cassandra 0.8.6 cluster with 2 nodes (in 2 DC's), RF = 2. > Actual data on the nodes is only 1GB. Disk latency < 1ms. Disk throughput ~ > 0.4MB/s. OS load always below 1 (on a 8 core machine with 16GB ram). > > When I'm running my writes against the cluster with cl = ONE all reads appear > to be faster then the writes. > > Average write speed = 1600us/operation > Average read speed = 200us/operation > > I'm really wondering why this is the case. Anyone got a clue? > > With kind regards, > Robin > > > > > > > > >
