I turned off 1 large cronjob which caused the CPU not to get used for ~ 60% once every 10 minutes. Both write and read are fast now. Just think I was overloading the node.
Weird though that shutting down the node did not improve the speed. Thank you all for your time! Robin 2012/1/5 aaron morton <[email protected]> > What happens when you turn off the cron jobs ? > > Cheers > > ----------------- > Aaron Morton > Freelance Developer > @aaronmorton > http://www.thelastpickle.com > > On 6/01/2012, at 6:57 AM, Philippe wrote: > > Unless you are doing huge batches no... don't have any other idea for > now... > > 2012/1/5 R. Verlangen <[email protected]> > >> The write and read load is very minimal the moment. Roughly 10 writes + >> 10 reads / second. So 20 operations per second. Don't think that overloads >> my cluster, does it? >> >> >> 2012/1/5 Philippe <[email protected]> >> >>> You may be overloading the cluster though... >>> >>> My hypothesis is that your traffic is being spread across your node and >>> that one slow node is slowing down the fraction of traffic that goes to >>> that node (when it's acting as coordinator). >>> So what I would do is reduce the read load a lot to make sure I don't >>> overload the cluster and measure if I see a 1/RF improvement in response >>> time which would validate my hypothesis. >>> >>> >>> 2012/1/5 R. Verlangen <[email protected]> >>> >>> It does not appear to affect the response time, certainly not in a >>>> positive way. >>>> >>>> >>>> 2012/1/5 Philippe <[email protected]> >>>> >>>>> What if you shutdown the cassandra service on the slow node, does that >>>>> improve your read performance ? >>>>> If it does then that sole node is responsible for the slow down >>>>> because it can't act as a coordinator fast enough. >>>>> >>>>> 2012/1/5 R. Verlangen <[email protected]> >>>>> >>>>> I'm also reading with CL = ONE >>>>>> >>>>>> >>>>>> 2012/1/5 Philippe <[email protected]> >>>>>> >>>>>>> Depending on the CL you're reading at it will yes : if the CL >>>>>>> requires that the "slow" node create a digest of the data and send it to >>>>>>> the coordinator then it might explain the poor performance on reads. >>>>>>> What >>>>>>> is your read CL ? >>>>>>> >>>>>>> 2012/1/5 R. Verlangen <[email protected]> >>>>>>> >>>>>>> As I posted this I noticed that the other node's CPU is running high >>>>>>>> on some other cronjobs (every couple of minutes to 60% usage). Is the >>>>>>>> lack >>>>>>>> of more CPU cycles a problem in this case? >>>>>>>> >>>>>>>> Robin >>>>>>>> >>>>>>>> 2012/1/5 R. Verlangen <[email protected]> >>>>>>>> >>>>>>>> CPU is idle (< 10% usage). Disk reads occasionally blocks over >>>>>>>>> 32/64K. Writes around 0-5MB per second. Network traffic 0.1 / 0.1 >>>>>>>>> MB/s (in >>>>>>>>> / out). Paging 0. System int ~ 1300, csw ~ 2500. >>>>>>>>> >>>>>>>>> >>>>>>>>> 2012/1/5 Philippe <[email protected]> >>>>>>>>> >>>>>>>>>> What can you see in vmstat/dstat ? >>>>>>>>>> Le 5 janv. 2012 11:58, "R. Verlangen" <[email protected]> a écrit : >>>>>>>>>> >>>>>>>>>> Hi there, >>>>>>>>>>> >>>>>>>>>>> I'm running a cassandra 0.8.6 cluster with 2 nodes (in 2 DC's), >>>>>>>>>>> RF = 2. Actual data on the nodes is only 1GB. Disk latency < 1ms. >>>>>>>>>>> Disk >>>>>>>>>>> throughput ~ 0.4MB/s. OS load always below 1 (on a 8 core machine >>>>>>>>>>> with 16GB >>>>>>>>>>> ram). >>>>>>>>>>> >>>>>>>>>>> When I'm running my writes against the cluster with cl = ONE all >>>>>>>>>>> reads appear to be faster then the writes. >>>>>>>>>>> >>>>>>>>>>> Average write speed = 1600us/operation >>>>>>>>>>> Average read speed = 200us/operation >>>>>>>>>>> >>>>>>>>>>> I'm really wondering why this is the case. Anyone got a clue? >>>>>>>>>>> >>>>>>>>>>> With kind regards, >>>>>>>>>>> Robin >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> > >
