Re: Debugging high tail read latencies (internal timeout)

2016-07-07 Thread daemeon reiydelle
Hmm. Would you mind looking at your network interface (appropriate netstat commands). if I am right you will be seeing packet errors, drops, retries, packet out of window receives, etc. What you may be missing is that you reported zero DROPPED latency. Not mean LATENCY. Check your netstats. ANY

Re: Debugging high tail read latencies (internal timeout)

2016-07-07 Thread Bryan Cheng
Hi Nimi, My suspicions would probably lie somewhere between GC and large partitions. The first tool would probably be a trace but if you experience full client timeouts from dropped messages you may find it hard to find the issue. You can try running the trace with cqlsh's timeouts cranked all

Debugging high tail read latencies (internal timeout)

2016-07-06 Thread Nimi Wariboko Jr
Hi, I've begun experiencing very high tail latencies across my clusters. While Cassandra's internal metrics report <1ms read latencies, measuring responses from within the driver in my applications (roundtrips of query/execute frames), have 90% round trip times of up to a second for very basic