Are the clocks synchronized across the cluster - probably, but I thought I would ask :)
On Wed, Oct 21, 2015 at 3:35 AM, Brice Figureau < brice+cassan...@daysofwonder.com> wrote: > Hi, > > On 20/10/2015 19:48, Carlos Alonso wrote: > > I think also having the output of cfhistograms could help. I'd like to > > know how many sstables are being hit during reads. > > Ooops, my bad I thought you mentioned proxyhistograms. > Here's the worst node cfhistograms: > > nodetool cfhistograms akka messages > akka/messages histograms > Percentile SSTables Write Latency Read Latency Partition > Size Cell Count > (micros) (micros) (bytes) > 50% 1.00 60.00 310.00 > 535 1 > 75% 2.00 103.00 9887.00 > 11864 17 > 95% 3.00 14237.00 126934.00 > 73457 215 > 98% 3.00 51012.00 219342.00 > 263210 446 > 99% 3.00 61214.00 1131752.00 > 454826 924 > Min 0.00 6.00 18.00 > 73 0 > Max 6.00 263210.00 5839588.00 > 10090808 1109 > > > > Also, which CL are you reading with? > > For the test request in cqlsh, I'm using ONE, but the application in which > I first observed the issue was using QUORUM. > > > cfstats is a local command, so maybe that node you've printed is working > > fine but there's another that is causing the latency. Can you check that > > command in all nodes? > > I checked and on all nodes, the read latency and read local latency are > within 15 to 40ms. > > I also noticed that C* was taking a fair bit of CPU on some of the nodes > (ranging from 60% to 200%), looking at ttop output it was mostly taken by > SharedPool-Worker threads, which I assume are the thread that are doing the > real query work. > > Well, I'm puzzled, and I'll keep searching, thanks for your help! > -- > Brice Figureau >
