Re: Cassandra p95 latencies

2023-08-25 Thread Andrew Weaver
Do you have the SSTables per read metric for before and after you increased the key cache size? If it was high before, that may have been the culprit meaning compaction tuning is in order. On Fri, Aug 25, 2023, 12:35 PM Shaurya Gupta wrote: > Thanks everyone. > Updating this thread - > We increa

Re: Cassandra p95 latencies

2023-08-25 Thread Shaurya Gupta
Thanks everyone. Updating this thread - We increased the key cache size from 100 MB to 200 MB and we believe that has brought down the latency from 40 ms p95 to 6 ms p95. I think there is still scope for improvement as both writes and reads are presently at p95 6 ms. I would expect writes to be low

Re: Cassandra p95 latencies

2023-08-14 Thread Elliott Sims via user
1. Check for Nagle/delayed-ack, but probably nodelay is getting set by the driver so it shouldn't be a problem. 2. Check for network latency (just regular old ping among hosts, during traffic) 3. Check your GC metrics and see if garbage collections line up with outliers. Some tuning can help th

Re: Cassandra p95 latencies

2023-08-14 Thread Josh McKenzie
> The queries are rightly designed Data modeling in Cassandra is 100% gray space; there unfortunately is no right or wrong design. You'll need to share basic shapes / contours of your data model for other folks to help you; seemingly innocuous things in a data model can cause unexpected issues w

Re: Cassandra p95 latencies

2023-08-11 Thread Jeff Jirsa
You’re going to have to help us help you 4.0 is pretty widely deployed. I’m not aware of a perf regression Can you give us a schema (anonymized) and queries and show us a trace ? On Aug 10, 2023, at 10:18 PM, Shaurya Gupta wrote:The queries are rightly designed as I already explained. 40 ms is wa

RE: Cassandra p95 latencies

2023-08-11 Thread Durity, Sean R via user
I would expect single digit ms latency on reads and writes. However, we have not done any performance testing on Apache Cassandra 4.x. Sean R. Durity INTERNAL USE From: Shaurya Gupta Sent: Friday, August 11, 2023 1:16 AM To: user@cassandra.apache.org Subject: [EXTERNAL] Re: Cassandra p95

Re: Cassandra p95 latencies

2023-08-10 Thread Shaurya Gupta
The queries are rightly designed as I already explained. 40 ms is way too high as compared to what I seen with other DBs and many a times with Cassandra 3.x versions. CPU consumed as I mentioned is not high, it is around 20%. On Thu, Aug 10, 2023 at 5:14 PM MyWorld wrote: > Hi, > P95 should not

Re: Cassandra p95 latencies

2023-08-10 Thread Abe Ratnofsky
40ms is definitely higher than expected. Have you run your queries with TRACING enabled to see where the latency is coming from?https://docs.datastax.com/en/cql-oss/3.3/cql/cql_reference/cqlshTracing.html40ms is also a fairly specific duration: https://eklitzke.org/the-caveats-of-tcp-nodelay> On Li

Re: Cassandra p95 latencies

2023-08-10 Thread MyWorld
Hi, P95 should not be a problem if rightly designed. Levelled compaction strategy further reduces this, however it consume some resources. For read, caching is also helpful. Can you check your cpu iowait as it could be the reason for delay Regards, Ashish On Fri, 11 Aug, 2023, 04:58 Shaurya Gupta