Re: ReadStage filling up and leading to Read Timeouts

2019-02-05 Thread Rajsekhar Mallick
Thank you Jeff for the link. Please do comment on the G1GC settings,if they are ok for the cluster. Also comment on reducing the concurrent reads to 32 on all nodes in the cluster. As has earlier lead to reads getting dropped. Will adding nodes to the cluster be helpful. Thanks, Rajsekhar Mallick

Re: ReadStage filling up and leading to Read Timeouts

2019-02-05 Thread Jeff Jirsa
https://docs.datastax.com/en/developer/java-driver/3.2/manual/paging/ -- Jeff Jirsa > On Feb 5, 2019, at 11:33 PM, Rajsekhar Mallick > wrote: > > Hello Jeff, > > Thanks for the reply. > We do have GC logs enabled. > We do observe gc pauses upto 2 seconds but quite often we see this issue

Re: ReadStage filling up and leading to Read Timeouts

2019-02-05 Thread Rajsekhar Mallick
Hello Jeff, Thanks for the reply. We do have GC logs enabled. We do observe gc pauses upto 2 seconds but quite often we see this issue even when the gc log reads good and clear. JVM Flags related to G1GC: Xms: 48G Xmx:48G Maxgcpausemillis=200 Parallels gc threads=32 Concurrent gc threads= 10

Re: ReadStage filling up and leading to Read Timeouts

2019-02-05 Thread Jeff Jirsa
What you're potentially seeing is the GC impact of reading a large partition - do you have GC logs or StatusLogger output indicating you're pausing? What are you actual JVM flags you're using? Given your heap size, the easiest mitigation may be significantly increasing your key cache size (up to

ReadStage filling up and leading to Read Timeouts

2019-02-05 Thread Rajsekhar Mallick
Hello Team, Cluster Details: 1. Number of Nodes in cluster : 7 2. Number of CPU cores: 48 3. Swap is enabled on all nodes 4. Memory available on all nodes : 120GB 5. Disk space available : 745GB 6. Cassandra version: 2.1 7. Active tables are using size-tiered compaction strategy 8. Read