Might be better to disable explicit gcs so the full gcs don’t even occur. It’s 
likely from the rmi dgc or directbytebuffers not any actual need to do gcs or 
the concurrent gc threads would be an issue as well.

Nodetool also has no excuse to use that big of a heap so it should have max 
size capped too (along with parallel and concurrent gc threads).

Chris

Sent from my iPhone

> On May 29, 2018, at 4:42 PM, kurt greaves <k...@instaclustr.com> wrote:
> 
> Good to know. So that confirms it's just the GC threads causing problems.
> 
>> On Tue., 29 May 2018, 22:02 Steinmaurer, Thomas, 
>> <thomas.steinmau...@dynatrace.com> wrote:
>> Kurt,
>> 
>>  
>> 
>> in our test it also didn’t made a difference with the default number of GC 
>> Threads (43 on our large machine) and running with Xmx128M or XmX31G 
>> (derived from $MAX_HEAP_SIZE). For both Xmx, we saw the high CPU caused by 
>> nodetool.
>> 
>>  
>> 
>> Regards,
>> 
>> Thomas
>> 
>>  
>> 
>> From: kurt greaves [mailto:k...@instaclustr.com] 
>> Sent: Dienstag, 29. Mai 2018 13:06
>> To: User <user@cassandra.apache.org>
>> Subject: Re: nodetool (2.1.18) - Xmx, ParallelGCThreads, High CPU usage
>> 
>>  
>> 
>> Thanks Thomas. After a bit more research today I found that the whole 
>> $MAX_HEAP_SIZE issue isn't really a problem because we don't explicitly set 
>> -Xms so the minimum heapsize by default will be 256mb, which isn't hugely 
>> problematic, and it's unlikely more than that would get allocated.
>> 
>>  
>> 
>> On 29 May 2018 at 09:29, Steinmaurer, Thomas 
>> <thomas.steinmau...@dynatrace.com> wrote:
>> 
>> Hi Kurt,
>> 
>>  
>> 
>> thanks for pointing me to the Xmx issue.
>> 
>>  
>> 
>> JIRA + patch (for Linux only based on C* 3.11) for the parallel GC thread 
>> issue is available here: 
>> https://issues.apache.org/jira/browse/CASSANDRA-14475
>> 
>>  
>> 
>> Thanks,
>> 
>> Thomas
>> 
>>  
>> 
>> From: kurt greaves [mailto:k...@instaclustr.com] 
>> Sent: Dienstag, 29. Mai 2018 05:54
>> To: User <user@cassandra.apache.org>
>> Subject: Re: nodetool (2.1.18) - Xmx, ParallelGCThreads, High CPU usage
>> 
>>  
>> 
>> 1) nodetool is reusing the $MAX_HEAP_SIZE environment variable, thus if we 
>> are running Cassandra with e.g. Xmx31G, nodetool is started with Xmx31G as 
>> well
>> 
>> This was fixed in 3.0.11/3.10 in CASSANDRA-12739. Not sure why it didn't 
>> make it into 2.1/2.2.
>> 
>> 2) As -XX:ParallelGCThreads is not explicitly set upon startup, this 
>> basically defaults to a value dependent on the number of cores. In our case, 
>> with the machine above, the number of parallel GC threads for the JVM is set 
>> to 43!
>> 3) Test-wise, we have adapted the nodetool startup script in a way to get a 
>> Java Flight Recording file on JVM exit, thus with each nodetool invocation 
>> we can inspect a JFR file. Here we may have seen System.gc() calls (without 
>> visible knowledge where they come from), GC times for the entire JVM 
>> life-time (e.g. ~1min) showing high cpu. This happened for both Xmx128M 
>> (default as it seems) and Xmx31G
>>  
>> After explicitly setting -XX:ParallelGCThreads=1 in the nodetool startup 
>> script, CPU usage spikes by nodetool are entirely gone.
>>  
>> Is this something which has been already adapted/tackled in Cassandra 
>> versions > 2.1 or worth to be considered as some sort of RFC?
>> 
>> Can you create a JIRA for this (and a patch, if you like)? We should be 
>> explicitly setting this on nodetool invocations.
>> 
>> ​
>> 
>> The contents of this e-mail are intended for the named addressee only. It 
>> contains information that may be confidential. Unless you are the named 
>> addressee or an authorized designee, you may not copy or use it, or disclose 
>> it to anyone else. If you received it in error please notify us immediately 
>> and then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) 
>> is a company registered in Linz whose registered office is at 4040 Linz, 
>> Austria, Freistädterstraße 313
>> 
>>  
>> 
>> The contents of this e-mail are intended for the named addressee only. It 
>> contains information that may be confidential. Unless you are the named 
>> addressee or an authorized designee, you may not copy or use it, or disclose 
>> it to anyone else. If you received it in error please notify us immediately 
>> and then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) 
>> is a company registered in Linz whose registered office is at 4040 Linz, 
>> Austria, Freistädterstraße 313

Reply via email to