High GC pauses leading to client seeing impact
Hello Team, I have a cluster of 17 nodes in production.(8 and 9 nodes in 2 DC). Cassandra version: 2.0.11 Client connecting using thrift over port 9160 Jdk version : 1.8.066 GC used : G1GC (16GB heap) Other GC settings: Maxgcpausemillis=200 Parallels gc threads=32 Concurrent gc threads= 10 Initiatingheapoccupancypercent=50 Number of cpu cores for each system : 40 Memory size: 185 GB Read/sec : 300 /sec on each node Writes/sec : 300/sec on each node Compaction strategy used : Size tiered compaction strategy Identified issues in the cluster: 1. Disk space usage across all nodes in the cluster is 80%. We are currently working on adding more storage on each node 2. There are 2 tables for which we keep on seeing large number of tombstones. One of table has read requests seeing 120 tombstones cells in last 5 mins as compared to 4 live cells. Tombstone warns and Error messages of query getting aborted is also seen. Current issue sen: 1. We keep on seeing GC pauses of few minutes randomly across nodes in the cluster. GC pauses of 120 seconds, even 770 seconds are also seen. 2. This leads to nodes getting stalled and client seeing direct impact 3. The GC pause we see, are not during any of G1GC phases. The GC log message prints “Time to stop threads took 770 seconds”. So it is not the garbage collector doing any work but stopping the threads at a safe point is taking so much of time. 4. This issue has surfaced recently after we changed 8GB(CMS) to 16GB(G1GC) across all nodes in the cluster. Kindly do help on the above issue. I am not able to exactly understand if the GC is wrongly tuned, other if this is something else. Thanks, Rajsekhar Mallick - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
Re: Cassandra.log
It will be part of cassandra startup script, of you are using RHEL its located at /etc/inti.d/cassandra Regards Sri Rathan On Sun, Feb 10, 2019, 2:45 PM Rahul Reddy wrote: > Hello, > > I'm using Cassandra 3.11.1 and trying to change the name of > cassandra.log(which is generate during startup of Cassandra) file. Can > someone point me to configuration file where it is configured . > System.log/debug.log located in logback.xml but not the Cassandra.log. > > > Thanks >
Cassandra.log
Hello, I'm using Cassandra 3.11.1 and trying to change the name of cassandra.log(which is generate during startup of Cassandra) file. Can someone point me to configuration file where it is configured . System.log/debug.log located in logback.xml but not the Cassandra.log. Thanks
Read and write trasanction per sec
Hi Is there a way to calculate or mesaure the read/ sec and write / sec in cassandra? I have prometheus tool to capture metrics it has read count ans write count not sure if its relevant to the clusster capacity to measure the read n write sec processing.
RE: Maximum memory usage
Can we the see “nodetool tablestats” for the biggest table as well. From: Kenneth Brotman [mailto:kenbrot...@yahoo.com.INVALID] Sent: Sunday, February 10, 2019 7:21 AM To: user@cassandra.apache.org Subject: RE: Maximum memory usage Okay, that’s at the moment it was calculated. Still need to see histograms. From: Rahul Reddy [mailto:rahulreddy1...@gmail.com] Sent: Sunday, February 10, 2019 7:09 AM To: user@cassandra.apache.org Subject: Re: Maximum memory usage Thanks Kenneth, 110mb is the biggest partition in our db On Sun, Feb 10, 2019, 9:55 AM Kenneth Brotman wrote: Rahul, Those partitions are tiny. Could you give us the table histograms for the biggest tables. Thanks, Kenneth Brotman From: Rahul Reddy [mailto:rahulreddy1...@gmail.com] Sent: Sunday, February 10, 2019 6:43 AM To: user@cassandra.apache.org Subject: Re: Maximum memory usage ```Percentile SSTables Write Latency Read LatencyPartition Size Cell Count (micros) (micros) (bytes) 50% 1.00 24.60219.34 258 4 75% 100 24.60379.02 446 8 95% 1.00 35.43379.02 924 17 98% 1.00 51.01379.02 1109 24 99% 1.00 61.21379.02 1331 29 Min 0.00 8.24126.94 104 0 Max 1.00152.32379.02 8409007 152321 ``` On Wed, Feb 6, 2019, 8:34 PM Kenneth Brotman wrote: Can you give us the “nodetool tablehistograms” Kenneth Brotman From: Rahul Reddy [mailto:rahulreddy1...@gmail.com] Sent: Wednesday, February 06, 2019 6:19 AM To: user@cassandra.apache.org Subject: Maximum memory usage Hello, I see maximum memory usage alerts in my system.log couple of times in a day as INFO. So far I haven't seen any issue with db. Why those messages are logged in system.log do we have any impact for reads/writes with those warnings? And what nerd to be looked INFO [RMI TCP Connection(170917)-127.0.0.1] 2019-02-05 23:15:47,408 NoSpamLogger.java:91 - Maximum memory usage reached (512.000MiB), cannot allocate chunk of 1.000MiB Thanks in advance
RE: Maximum memory usage
Okay, that’s at the moment it was calculated. Still need to see histograms. From: Rahul Reddy [mailto:rahulreddy1...@gmail.com] Sent: Sunday, February 10, 2019 7:09 AM To: user@cassandra.apache.org Subject: Re: Maximum memory usage Thanks Kenneth, 110mb is the biggest partition in our db On Sun, Feb 10, 2019, 9:55 AM Kenneth Brotman wrote: Rahul, Those partitions are tiny. Could you give us the table histograms for the biggest tables. Thanks, Kenneth Brotman From: Rahul Reddy [mailto:rahulreddy1...@gmail.com] Sent: Sunday, February 10, 2019 6:43 AM To: user@cassandra.apache.org Subject: Re: Maximum memory usage ```Percentile SSTables Write Latency Read LatencyPartition Size Cell Count (micros) (micros) (bytes) 50% 1.00 24.60219.34 258 4 75% 100 24.60379.02 446 8 95% 1.00 35.43379.02 924 17 98% 1.00 51.01379.02 1109 24 99% 1.00 61.21379.02 1331 29 Min 0.00 8.24126.94 104 0 Max 1.00152.32379.02 8409007 152321 ``` On Wed, Feb 6, 2019, 8:34 PM Kenneth Brotman wrote: Can you give us the “nodetool tablehistograms” Kenneth Brotman From: Rahul Reddy [mailto:rahulreddy1...@gmail.com] Sent: Wednesday, February 06, 2019 6:19 AM To: user@cassandra.apache.org Subject: Maximum memory usage Hello, I see maximum memory usage alerts in my system.log couple of times in a day as INFO. So far I haven't seen any issue with db. Why those messages are logged in system.log do we have any impact for reads/writes with those warnings? And what nerd to be looked INFO [RMI TCP Connection(170917)-127.0.0.1] 2019-02-05 23:15:47,408 NoSpamLogger.java:91 - Maximum memory usage reached (512.000MiB), cannot allocate chunk of 1.000MiB Thanks in advance
Re: Maximum memory usage
Thanks Kenneth, 110mb is the biggest partition in our db On Sun, Feb 10, 2019, 9:55 AM Kenneth Brotman wrote: > Rahul, > > > > Those partitions are tiny. Could you give us the table histograms for the > biggest tables. > > > > Thanks, > > > > Kenneth Brotman > > > > *From:* Rahul Reddy [mailto:rahulreddy1...@gmail.com] > *Sent:* Sunday, February 10, 2019 6:43 AM > *To:* user@cassandra.apache.org > *Subject:* Re: Maximum memory usage > > > > ```Percentile SSTables Write Latency Read LatencyPartition > SizeCell Count > > (micros) (micros) > (bytes) > > 50% 1.00 24.60219.34 > 258 4 > > 75% 1.00 24.60379.02 > 446 8 > > 95% 1.00 35.43379.02 > 92417 > > 98% 1.00 51.01379.02 > 110924 > > 99% 1.00 61.21379.02 > 133129 > > Min 0.00 8.24126.94 > 104 0 > > Max 1.00152.32379.02 > 8409007152321 > > ``` > > > > On Wed, Feb 6, 2019, 8:34 PM Kenneth Brotman > wrote: > > Can you give us the “nodetool tablehistograms” > > > > Kenneth Brotman > > > > *From:* Rahul Reddy [mailto:rahulreddy1...@gmail.com] > *Sent:* Wednesday, February 06, 2019 6:19 AM > *To:* user@cassandra.apache.org > *Subject:* Maximum memory usage > > > > Hello, > > > > I see maximum memory usage alerts in my system.log couple of times in a > day as INFO. So far I haven't seen any issue with db. Why those messages > are logged in system.log do we have any impact for reads/writes with those > warnings? And what nerd to be looked > > > > INFO [RMI TCP Connection(170917)-127.0.0.1] 2019-02-05 23:15:47,408 > NoSpamLogger.java:91 - Maximum memory usage reached (512.000MiB), cannot > allocate chunk of 1.000MiB > > > > Thanks in advance > >
Re: Maximum memory usage
One of the other db with 100mb partition* out of memory happens very frequently. ```Percentile SSTables Write Latency Read LatencyPartition SizeCell Count (micros) (micros) (bytes) 50% 0.00 0.00 0.00182785 2759 75% 0.00 0.00 0.00943127 14237 95% 0.00 0.00 0.00 4866323 73457 98% 0.00 0.00 0.00 12108970 152321 99% 0.00 0.00 0.00 17436917 219342 Min 0.00 0.00 0.00 150 2 Max 0.00 0.00 0.00 107964792 1358102 On Thu, Feb 7, 2019, 2:29 AM dinesh.jo...@yahoo.com.INVALID wrote: > Are you running any nodetool commands during that period? IIRC, this is a > log entry emitted by the BufferPool. It may be harm unless it's happening > very often or logging a OOM. > > Dinesh > > > On Wednesday, February 6, 2019, 6:19:42 AM PST, Rahul Reddy < > rahulreddy1...@gmail.com> wrote: > > > Hello, > > I see maximum memory usage alerts in my system.log couple of times in a > day as INFO. So far I haven't seen any issue with db. Why those messages > are logged in system.log do we have any impact for reads/writes with those > warnings? And what nerd to be looked > > INFO [RMI TCP Connection(170917)-127.0.0.1] 2019-02-05 23:15:47,408 > NoSpamLogger.java:91 - Maximum memory usage reached (512.000MiB), cannot > allocate chunk of 1.000MiB > > Thanks in advance >
RE: Maximum memory usage
Rahul, Those partitions are tiny. Could you give us the table histograms for the biggest tables. Thanks, Kenneth Brotman From: Rahul Reddy [mailto:rahulreddy1...@gmail.com] Sent: Sunday, February 10, 2019 6:43 AM To: user@cassandra.apache.org Subject: Re: Maximum memory usage ```Percentile SSTables Write Latency Read LatencyPartition Size Cell Count (micros) (micros) (bytes) 50% 1.00 24.60219.34 258 4 75% 1.00 24.60379.02 446 8 95% 1.00 35.43379.02 924 17 98% 1.00 51.01379.02 1109 24 99% 1.00 61.21379.02 1331 29 Min 0.00 8.24126.94 104 0 Max 1.00152.32379.02 8409007 152321 ``` On Wed, Feb 6, 2019, 8:34 PM Kenneth Brotman wrote: Can you give us the “nodetool tablehistograms” Kenneth Brotman From: Rahul Reddy [mailto:rahulreddy1...@gmail.com] Sent: Wednesday, February 06, 2019 6:19 AM To: user@cassandra.apache.org Subject: Maximum memory usage Hello, I see maximum memory usage alerts in my system.log couple of times in a day as INFO. So far I haven't seen any issue with db. Why those messages are logged in system.log do we have any impact for reads/writes with those warnings? And what nerd to be looked INFO [RMI TCP Connection(170917)-127.0.0.1] 2019-02-05 23:15:47,408 NoSpamLogger.java:91 - Maximum memory usage reached (512.000MiB), cannot allocate chunk of 1.000MiB Thanks in advance
Re: Maximum memory usage
No not running any nodetool commands. It happens 2 to 3 times a day On Thu, Feb 7, 2019, 2:29 AM dinesh.jo...@yahoo.com.INVALID wrote: > Are you running any nodetool commands during that period? IIRC, this is a > log entry emitted by the BufferPool. It may be harm unless it's happening > very often or logging a OOM. > > Dinesh > > > On Wednesday, February 6, 2019, 6:19:42 AM PST, Rahul Reddy < > rahulreddy1...@gmail.com> wrote: > > > Hello, > > I see maximum memory usage alerts in my system.log couple of times in a > day as INFO. So far I haven't seen any issue with db. Why those messages > are logged in system.log do we have any impact for reads/writes with those > warnings? And what nerd to be looked > > INFO [RMI TCP Connection(170917)-127.0.0.1] 2019-02-05 23:15:47,408 > NoSpamLogger.java:91 - Maximum memory usage reached (512.000MiB), cannot > allocate chunk of 1.000MiB > > Thanks in advance >
Re: Maximum memory usage
```Percentile SSTables Write Latency Read LatencyPartition SizeCell Count (micros) (micros) (bytes) 50% 1.00 24.60219.34 258 4 75% 1.00 24.60379.02 446 8 95% 1.00 35.43379.02 924 17 98% 1.00 51.01379.02 1109 24 99% 1.00 61.21379.02 1331 29 Min 0.00 8.24126.94 104 0 Max 1.00152.32379.02 8409007 152321 ``` On Wed, Feb 6, 2019, 8:34 PM Kenneth Brotman wrote: > Can you give us the “nodetool tablehistograms” > > > > Kenneth Brotman > > > > *From:* Rahul Reddy [mailto:rahulreddy1...@gmail.com] > *Sent:* Wednesday, February 06, 2019 6:19 AM > *To:* user@cassandra.apache.org > *Subject:* Maximum memory usage > > > > Hello, > > > > I see maximum memory usage alerts in my system.log couple of times in a > day as INFO. So far I haven't seen any issue with db. Why those messages > are logged in system.log do we have any impact for reads/writes with those > warnings? And what nerd to be looked > > > > INFO [RMI TCP Connection(170917)-127.0.0.1] 2019-02-05 23:15:47,408 > NoSpamLogger.java:91 - Maximum memory usage reached (512.000MiB), cannot > allocate chunk of 1.000MiB > > > > Thanks in advance >
