The spikes in latency don’t seem to be correlated to an increase in reads.
The cluster’s workload is usually handling a maximum workload of 4200
reads/sec per node, with writes being significantly less, at ~200/sec per
node. Usually it will be fine with this, with read latencies at around
/**
* Verbs it's okay to drop if the request has been queued longer than
the request timeout. These
* all correspond to client requests or something triggered by them; we
don't want to
* drop internal messages like bootstrap or repair notifications.
*/
public static
Hi, could it be due to having noisy neighbour? Do you have graphs
statistics ping between nodes?
Jason
On Mon, Jan 6, 2014 at 7:28 AM, Blake Eggleston bl...@shift.com wrote:
Hi,
I’ve been having a problem with 3 neighboring nodes in our cluster having
their read latencies jump up to 9000ms
That’s a good point. CPU steal time is very low, but I haven’t observed
internode ping times during one of the peaks, I’ll have to check that out.
Another thing I’ve noticed is that cassandra starts dropping read messages
during the spikes, as reported by tpstats. This indicates that there’s
Hi,
I’ve been having a problem with 3 neighboring nodes in our cluster having their
read latencies jump up to 9000ms - 18000ms for a few minutes (as reported by
opscenter), then come back down.
We’re running a 6 node cluster, on AWS hi1.4xlarge instances, with cassandra
reading and writing to
