Re: nodetool (2.1.18) - Xmx, ParallelGCThreads, High CPU usage

[Chris Lohfink]

Might be better to disable explicit gcs so the full gcs don’t even occur. It’s 
likely from the rmi dgc or directbytebuffers not any actual need to do gcs or 
the concurrent gc threads would be an issue as well.

Nodetool also has no excuse to use that big of a heap so it should have max 
size capped too (along with parallel and concurrent gc threads).

Chris

Sent from my iPhone

> On May 29, 2018, at 4:42 PM, kurt greaves  wrote:
> 
> Good to know. So that confirms it's just the GC threads causing problems.
> 
>> On Tue., 29 May 2018, 22:02 Steinmaurer, Thomas, 
>>  wrote:
>> Kurt,
>> 
>>  
>> 
>> in our test it also didn’t made a difference with the default number of GC 
>> Threads (43 on our large machine) and running with Xmx128M or XmX31G 
>> (derived from $MAX_HEAP_SIZE). For both Xmx, we saw the high CPU caused by 
>> nodetool.
>> 
>>  
>> 
>> Regards,
>> 
>> Thomas
>> 
>>  
>> 
>> From: kurt greaves [mailto:k...@instaclustr.com] 
>> Sent: Dienstag, 29. Mai 2018 13:06
>> To: User 
>> Subject: Re: nodetool (2.1.18) - Xmx, ParallelGCThreads, High CPU usage
>> 
>>  
>> 
>> Thanks Thomas. After a bit more research today I found that the whole 
>> $MAX_HEAP_SIZE issue isn't really a problem because we don't explicitly set 
>> -Xms so the minimum heapsize by default will be 256mb, which isn't hugely 
>> problematic, and it's unlikely more than that would get allocated.
>> 
>>  
>> 
>> On 29 May 2018 at 09:29, Steinmaurer, Thomas 
>>  wrote:
>> 
>> Hi Kurt,
>> 
>>  
>> 
>> thanks for pointing me to the Xmx issue.
>> 
>>  
>> 
>> JIRA + patch (for Linux only based on C* 3.11) for the parallel GC thread 
>> issue is available here: 
>> https://issues.apache.org/jira/browse/CASSANDRA-14475
>> 
>>  
>> 
>> Thanks,
>> 
>> Thomas
>> 
>>  
>> 
>> From: kurt greaves [mailto:k...@instaclustr.com] 
>> Sent: Dienstag, 29. Mai 2018 05:54
>> To: User 
>> Subject: Re: nodetool (2.1.18) - Xmx, ParallelGCThreads, High CPU usage
>> 
>>  
>> 
>> 1) nodetool is reusing the $MAX_HEAP_SIZE environment variable, thus if we 
>> are running Cassandra with e.g. Xmx31G, nodetool is started with Xmx31G as 
>> well
>> 
>> This was fixed in 3.0.11/3.10 in CASSANDRA-12739. Not sure why it didn't 
>> make it into 2.1/2.2.
>> 
>> 2) As -XX:ParallelGCThreads is not explicitly set upon startup, this 
>> basically defaults to a value dependent on the number of cores. In our case, 
>> with the machine above, the number of parallel GC threads for the JVM is set 
>> to 43!
>> 3) Test-wise, we have adapted the nodetool startup script in a way to get a 
>> Java Flight Recording file on JVM exit, thus with each nodetool invocation 
>> we can inspect a JFR file. Here we may have seen System.gc() calls (without 
>> visible knowledge where they come from), GC times for the entire JVM 
>> life-time (e.g. ~1min) showing high cpu. This happened for both Xmx128M 
>> (default as it seems) and Xmx31G
>>  
>> After explicitly setting -XX:ParallelGCThreads=1 in the nodetool startup 
>> script, CPU usage spikes by nodetool are entirely gone.
>>  
>> Is this something which has been already adapted/tackled in Cassandra 
>> versions > 2.1 or worth to be considered as some sort of RFC?
>> 
>> Can you create a JIRA for this (and a patch, if you like)? We should be 
>> explicitly setting this on nodetool invocations.
>> 
>> ​
>> 
>> The contents of this e-mail are intended for the named addressee only. It 
>> contains information that may be confidential. Unless you are the named 
>> addressee or an authorized designee, you may not copy or use it, or disclose 
>> it to anyone else. If you received it in error please notify us immediately 
>> and then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) 
>> is a company registered in Linz whose registered office is at 4040 Linz, 
>> Austria, Freistädterstraße 313
>> 
>>  
>> 
>> The contents of this e-mail are intended for the named addressee only. It 
>> contains information that may be confidential. Unless you are the named 
>> addressee or an authorized designee, you may not copy or use it, or disclose 
>> it to anyone else. If you received it in error please notify us immediately 
>> and then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) 
>> is a company registered in Linz whose registered office is at 4040 Linz, 
>> Austria, Freistädterstraße 313

Re: nodetool (2.1.18) - Xmx, ParallelGCThreads, High CPU usage

[kurt greaves]

Good to know. So that confirms it's just the GC threads causing problems.

On Tue., 29 May 2018, 22:02 Steinmaurer, Thomas, <
thomas.steinmau...@dynatrace.com> wrote:

> Kurt,
>
>
>
> in our test it also didn’t made a difference with the default number of GC
> Threads (43 on our large machine) and running with Xmx128M or XmX31G
> (derived from $MAX_HEAP_SIZE). For both Xmx, we saw the high CPU caused by
> nodetool.
>
>
>
> Regards,
>
> Thomas
>
>
>
> *From:* kurt greaves [mailto:k...@instaclustr.com]
> *Sent:* Dienstag, 29. Mai 2018 13:06
> *To:* User 
> *Subject:* Re: nodetool (2.1.18) - Xmx, ParallelGCThreads, High CPU usage
>
>
>
> Thanks Thomas. After a bit more research today I found that the whole
> $MAX_HEAP_SIZE issue isn't really a problem because we don't explicitly set
> -Xms so the minimum heapsize by default will be 256mb, which isn't hugely
> problematic, and it's unlikely more than that would get allocated.
>
>
>
> On 29 May 2018 at 09:29, Steinmaurer, Thomas <
> thomas.steinmau...@dynatrace.com> wrote:
>
> Hi Kurt,
>
>
>
> thanks for pointing me to the Xmx issue.
>
>
>
> JIRA + patch (for Linux only based on C* 3.11) for the parallel GC thread
> issue is available here:
> https://issues.apache.org/jira/browse/CASSANDRA-14475
>
>
>
> Thanks,
>
> Thomas
>
>
>
> *From:* kurt greaves [mailto:k...@instaclustr.com]
> *Sent:* Dienstag, 29. Mai 2018 05:54
> *To:* User 
> *Subject:* Re: nodetool (2.1.18) - Xmx, ParallelGCThreads, High CPU usage
>
>
>
> 1) nodetool is reusing the $MAX_HEAP_SIZE environment variable, thus if we
> are running Cassandra with e.g. Xmx31G, nodetool is started with Xmx31G as
> well
>
> This was fixed in 3.0.11/3.10 in CASSANDRA-12739
> <https://issues.apache.org/jira/browse/CASSANDRA-12739>. Not sure why it
> didn't make it into 2.1/2.2.
>
> 2) As -XX:ParallelGCThreads is not explicitly set upon startup, this
> basically defaults to a value dependent on the number of cores. In our
> case, with the machine above, the number of parallel GC threads for the JVM
> is set to 43!
> 3) Test-wise, we have adapted the nodetool startup script in a way to get
> a Java Flight Recording file on JVM exit, thus with each nodetool
> invocation we can inspect a JFR file. Here we may have seen System.gc()
> calls (without visible knowledge where they come from), GC times for the
> entire JVM life-time (e.g. ~1min) showing high cpu. This happened for both
> Xmx128M (default as it seems) and Xmx31G
>
> After explicitly setting -XX:ParallelGCThreads=1 in the nodetool startup
> script, CPU usage spikes by nodetool are entirely gone.
>
> Is this something which has been already adapted/tackled in Cassandra
> versions > 2.1 or worth to be considered as some sort of RFC?
>
> Can you create a JIRA for this (and a patch, if you like)? We should be
> explicitly setting this on nodetool invocations.
>
> ​
>
> The contents of this e-mail are intended for the named addressee only. It
> contains information that may be confidential. Unless you are the named
> addressee or an authorized designee, you may not copy or use it, or
> disclose it to anyone else. If you received it in error please notify us
> immediately and then destroy it. Dynatrace Austria GmbH (registration
> number FN 91482h) is a company registered in Linz whose registered office
> is at 4040 Linz, Austria, Freistädterstraße 313
>
>
> The contents of this e-mail are intended for the named addressee only. It
> contains information that may be confidential. Unless you are the named
> addressee or an authorized designee, you may not copy or use it, or
> disclose it to anyone else. If you received it in error please notify us
> immediately and then destroy it. Dynatrace Austria GmbH (registration
> number FN 91482h) is a company registered in Linz whose registered office
> is at 4040 Linz, Austria, Freistädterstraße 313
>

RE: nodetool (2.1.18) - Xmx, ParallelGCThreads, High CPU usage

[Steinmaurer, Thomas]

Kurt,

in our test it also didn’t made a difference with the default number of GC 
Threads (43 on our large machine) and running with Xmx128M or XmX31G (derived 
from $MAX_HEAP_SIZE). For both Xmx, we saw the high CPU caused by nodetool.

Regards,
Thomas

From: kurt greaves [mailto:k...@instaclustr.com]
Sent: Dienstag, 29. Mai 2018 13:06
To: User 
Subject: Re: nodetool (2.1.18) - Xmx, ParallelGCThreads, High CPU usage

Thanks Thomas. After a bit more research today I found that the whole 
$MAX_HEAP_SIZE issue isn't really a problem because we don't explicitly set 
-Xms so the minimum heapsize by default will be 256mb, which isn't hugely 
problematic, and it's unlikely more than that would get allocated.

On 29 May 2018 at 09:29, Steinmaurer, Thomas 
mailto:thomas.steinmau...@dynatrace.com>> 
wrote:
Hi Kurt,

thanks for pointing me to the Xmx issue.

JIRA + patch (for Linux only based on C* 3.11) for the parallel GC thread issue 
is available here: https://issues.apache.org/jira/browse/CASSANDRA-14475

Thanks,
Thomas

From: kurt greaves [mailto:k...@instaclustr.com<mailto:k...@instaclustr.com>]
Sent: Dienstag, 29. Mai 2018 05:54
To: User mailto:user@cassandra.apache.org>>
Subject: Re: nodetool (2.1.18) - Xmx, ParallelGCThreads, High CPU usage

1) nodetool is reusing the $MAX_HEAP_SIZE environment variable, thus if we are 
running Cassandra with e.g. Xmx31G, nodetool is started with Xmx31G as well
This was fixed in 3.0.11/3.10 in 
CASSANDRA-12739<https://issues.apache.org/jira/browse/CASSANDRA-12739>. Not 
sure why it didn't make it into 2.1/2.2.
2) As -XX:ParallelGCThreads is not explicitly set upon startup, this basically 
defaults to a value dependent on the number of cores. In our case, with the 
machine above, the number of parallel GC threads for the JVM is set to 43!
3) Test-wise, we have adapted the nodetool startup script in a way to get a 
Java Flight Recording file on JVM exit, thus with each nodetool invocation we 
can inspect a JFR file. Here we may have seen System.gc() calls (without 
visible knowledge where they come from), GC times for the entire JVM life-time 
(e.g. ~1min) showing high cpu. This happened for both Xmx128M (default as it 
seems) and Xmx31G

After explicitly setting -XX:ParallelGCThreads=1 in the nodetool startup 
script, CPU usage spikes by nodetool are entirely gone.

Is this something which has been already adapted/tackled in Cassandra versions 
> 2.1 or worth to be considered as some sort of RFC?
Can you create a JIRA for this (and a patch, if you like)? We should be 
explicitly setting this on nodetool invocations.
​
The contents of this e-mail are intended for the named addressee only. It 
contains information that may be confidential. Unless you are the named 
addressee or an authorized designee, you may not copy or use it, or disclose it 
to anyone else. If you received it in error please notify us immediately and 
then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a 
company registered in Linz whose registered office is at 4040 Linz, Austria, 
Freistädterstraße 313

The contents of this e-mail are intended for the named addressee only. It 
contains information that may be confidential. Unless you are the named 
addressee or an authorized designee, you may not copy or use it, or disclose it 
to anyone else. If you received it in error please notify us immediately and 
then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a 
company registered in Linz whose registered office is at 4040 Linz, Austria, 
Freistädterstraße 313

Re: nodetool (2.1.18) - Xmx, ParallelGCThreads, High CPU usage

[kurt greaves]

Thanks Thomas. After a bit more research today I found that the whole
$MAX_HEAP_SIZE issue isn't really a problem because we don't explicitly set
-Xms so the minimum heapsize by default will be 256mb, which isn't hugely
problematic, and it's unlikely more than that would get allocated.

On 29 May 2018 at 09:29, Steinmaurer, Thomas <
thomas.steinmau...@dynatrace.com> wrote:

> Hi Kurt,
>
>
>
> thanks for pointing me to the Xmx issue.
>
>
>
> JIRA + patch (for Linux only based on C* 3.11) for the parallel GC thread
> issue is available here: https://issues.apache.org/
> jira/browse/CASSANDRA-14475
>
>
>
> Thanks,
>
> Thomas
>
>
>
> *From:* kurt greaves [mailto:k...@instaclustr.com]
> *Sent:* Dienstag, 29. Mai 2018 05:54
> *To:* User 
> *Subject:* Re: nodetool (2.1.18) - Xmx, ParallelGCThreads, High CPU usage
>
>
>
> 1) nodetool is reusing the $MAX_HEAP_SIZE environment variable, thus if we
> are running Cassandra with e.g. Xmx31G, nodetool is started with Xmx31G as
> well
>
> This was fixed in 3.0.11/3.10 in CASSANDRA-12739
> <https://issues.apache.org/jira/browse/CASSANDRA-12739>. Not sure why it
> didn't make it into 2.1/2.2.
>
> 2) As -XX:ParallelGCThreads is not explicitly set upon startup, this
> basically defaults to a value dependent on the number of cores. In our
> case, with the machine above, the number of parallel GC threads for the JVM
> is set to 43!
> 3) Test-wise, we have adapted the nodetool startup script in a way to get
> a Java Flight Recording file on JVM exit, thus with each nodetool
> invocation we can inspect a JFR file. Here we may have seen System.gc()
> calls (without visible knowledge where they come from), GC times for the
> entire JVM life-time (e.g. ~1min) showing high cpu. This happened for both
> Xmx128M (default as it seems) and Xmx31G
>
> After explicitly setting -XX:ParallelGCThreads=1 in the nodetool startup
> script, CPU usage spikes by nodetool are entirely gone.
>
> Is this something which has been already adapted/tackled in Cassandra
> versions > 2.1 or worth to be considered as some sort of RFC?
>
> Can you create a JIRA for this (and a patch, if you like)? We should be
> explicitly setting this on nodetool invocations.
>
> ​
> The contents of this e-mail are intended for the named addressee only. It
> contains information that may be confidential. Unless you are the named
> addressee or an authorized designee, you may not copy or use it, or
> disclose it to anyone else. If you received it in error please notify us
> immediately and then destroy it. Dynatrace Austria GmbH (registration
> number FN 91482h) is a company registered in Linz whose registered office
> is at 4040 Linz, Austria, Freistädterstraße 313
>

RE: nodetool (2.1.18) - Xmx, ParallelGCThreads, High CPU usage

[Steinmaurer, Thomas]

Hi Kurt,

thanks for pointing me to the Xmx issue.

JIRA + patch (for Linux only based on C* 3.11) for the parallel GC thread issue 
is available here: https://issues.apache.org/jira/browse/CASSANDRA-14475

Thanks,
Thomas

From: kurt greaves [mailto:k...@instaclustr.com]
Sent: Dienstag, 29. Mai 2018 05:54
To: User 
Subject: Re: nodetool (2.1.18) - Xmx, ParallelGCThreads, High CPU usage

1) nodetool is reusing the $MAX_HEAP_SIZE environment variable, thus if we are 
running Cassandra with e.g. Xmx31G, nodetool is started with Xmx31G as well
This was fixed in 3.0.11/3.10 in 
CASSANDRA-12739<https://issues.apache.org/jira/browse/CASSANDRA-12739>. Not 
sure why it didn't make it into 2.1/2.2.
2) As -XX:ParallelGCThreads is not explicitly set upon startup, this basically 
defaults to a value dependent on the number of cores. In our case, with the 
machine above, the number of parallel GC threads for the JVM is set to 43!
3) Test-wise, we have adapted the nodetool startup script in a way to get a 
Java Flight Recording file on JVM exit, thus with each nodetool invocation we 
can inspect a JFR file. Here we may have seen System.gc() calls (without 
visible knowledge where they come from), GC times for the entire JVM life-time 
(e.g. ~1min) showing high cpu. This happened for both Xmx128M (default as it 
seems) and Xmx31G

After explicitly setting -XX:ParallelGCThreads=1 in the nodetool startup 
script, CPU usage spikes by nodetool are entirely gone.

Is this something which has been already adapted/tackled in Cassandra versions 
> 2.1 or worth to be considered as some sort of RFC?
Can you create a JIRA for this (and a patch, if you like)? We should be 
explicitly setting this on nodetool invocations.
​
The contents of this e-mail are intended for the named addressee only. It 
contains information that may be confidential. Unless you are the named 
addressee or an authorized designee, you may not copy or use it, or disclose it 
to anyone else. If you received it in error please notify us immediately and 
then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a 
company registered in Linz whose registered office is at 4040 Linz, Austria, 
Freistädterstraße 313

Re: nodetool (2.1.18) - Xmx, ParallelGCThreads, High CPU usage

[kurt greaves]

>
> 1) nodetool is reusing the $MAX_HEAP_SIZE environment variable, thus if we
> are running Cassandra with e.g. Xmx31G, nodetool is started with Xmx31G as
> well

This was fixed in 3.0.11/3.10 in CASSANDRA-12739
. Not sure why it
didn't make it into 2.1/2.2.

> 2) As -XX:ParallelGCThreads is not explicitly set upon startup, this
> basically defaults to a value dependent on the number of cores. In our
> case, with the machine above, the number of parallel GC threads for the JVM
> is set to 43!
> 3) Test-wise, we have adapted the nodetool startup script in a way to get
> a Java Flight Recording file on JVM exit, thus with each nodetool
> invocation we can inspect a JFR file. Here we may have seen System.gc()
> calls (without visible knowledge where they come from), GC times for the
> entire JVM life-time (e.g. ~1min) showing high cpu. This happened for both
> Xmx128M (default as it seems) and Xmx31G
>
> After explicitly setting -XX:ParallelGCThreads=1 in the nodetool startup
> script, CPU usage spikes by nodetool are entirely gone.
>
> Is this something which has been already adapted/tackled in Cassandra
> versions > 2.1 or worth to be considered as some sort of RFC?
>
Can you create a JIRA for this (and a patch, if you like)? We should be
explicitly setting this on nodetool invocations.
​