Re: R/W timeouts VS number of tables in keyspace

2021-07-22 Thread Erick Ramirez
I wanted to add a word of warning that switching to G1 won't necessarily give you breathing space. In fact, I know it definitely won't. In your original post, it looked like the node had a very small heap (2GB). In my experience, you need to allocate at least 8GB of memory to the heap for

Re: R/W timeouts VS number of tables in keyspace

2021-07-22 Thread Scott Hirleman
les, no matter which GC you use. >> >> >> >> Sean Durity – Staff Systems Engineer, Cassandra >> >> >> >> *From:* Luca Rondanini >> *Sent:* Monday, July 19, 2021 11:34 AM >> *To:* user@cassandra.apache.org >> *Subject:* [EXTERNAL]

Re: R/W timeouts VS number of tables in keyspace

2021-07-20 Thread Luca Rondanini
to engineer around the 900+ > tables, no matter which GC you use. > > > > Sean Durity – Staff Systems Engineer, Cassandra > > > > *From:* Luca Rondanini > *Sent:* Monday, July 19, 2021 11:34 AM > *To:* user@cassandra.apache.org > *Subject:* [EXTERNAL] R/W timeouts V

RE: Re: R/W timeouts VS number of tables in keyspace

2021-07-19 Thread MyWorld
Yes it seems like a GC issue. And since the default timeout for write/read is 2000ms and 5000ms, you might be experiencing timeout issues. But you need to check reason behind this GC pause. Heavy partition or high tombstones could be one reason. Check your table stats for the same. As suggested

Re: R/W timeouts VS number of tables in keyspace

2021-07-19 Thread Yakir Gibraltar
In order to tune GC, you need gc.log or jvm metrics, you can check on https://gceasy.io/ and see the results before and after the change. On Mon, Jul 19, 2021 at 7:21 PM Luca Rondanini wrote: > Thanks Yakir, > > I can already experience slow repairs and startups but I'd like to > stabilize the

Re: R/W timeouts VS number of tables in keyspace

2021-07-19 Thread Luca Rondanini
Thanks Yakir, I can already experience slow repairs and startups but I'd like to stabilize the system before jumping into refactoring (columns are not a problem, max 10/cols per table). Do you believe it's a GC problem to cause the timeouts and crashes? I'll give it a try and update this post.

Re: R/W timeouts VS number of tables in keyspace

2021-07-19 Thread Yakir Gibraltar
I recommend rethinking about this design, hard to maintain, slow startup and repair . About GC, try to replace CMS with G1 , see doc : https://docs.datastax.com/en/dse/6.0/dse-admin/datastax_enterprise/operations/opsTuningGcAbout.html BTW, also many columns may affect performance, see doc:

R/W timeouts VS number of tables in keyspace

2021-07-19 Thread Luca Rondanini
Hi all, I have a keyspace with almost 900 tables. Lately I started receiving lots of w/r timeouts (eg com.datastax.driver.core.exceptions.Read/WriteTimeoutException: Cassandra timeout during write query at consistency LOCAL_ONE (1 replica were required but only 0 acknowledged the write). *I'm