Re: RemoveNode CPU Spike Question

[Shalom Sagges]

Hi Anubhav,

This happened to us as well, on all nodes in the DC. We found that after
performing removenode, all other nodes suddenly started to do a lot of
compactions that increased CPU.
To mitigate that, we used nodetool disableautocompaction before removing
the node. Then, after removal, we slowly enabled autocompaction (a few
minutes between each enable) on the nodes one by one.
This helped with the CPU increase you've mentioned.



Shalom Sagges
DBA
T: +972-74-700-4035
 
 We Create Meaningful Connections



On Tue, Jan 10, 2017 at 8:03 PM, Anubhav Kale 
wrote:

> Well, looking through logs I confirmed that my understanding below is
> correct, but would be good to hear from experts for sure 
>
>
>
> *From:* Anubhav Kale [mailto:anubhav.k...@microsoft.com]
> *Sent:* Tuesday, January 10, 2017 9:58 AM
> *To:* user@cassandra.apache.org
> *Cc:* Sean Usher 
> *Subject:* RemoveNode CPU Spike Question
>
>
>
> Hello,
>
>
>
> Recently, I started noticing an interesting pattern. When I execute
> “removenode”, a subset of the nodes that now own the tokens result it in a
> CPU spike / disk activity, and sometimes SSTables on those nodes shoot up.
>
>
>
> After looking through the code, it appears to me that below function
> forces data to be streamed from some of the new nodes to the node from
> where “removenode” is kicked in. Is my understanding correct ?
>
>
>
> https://github.com/apache/cassandra/blob/d384e781d6f7c028dbe88cfe9dd3e9
> 66e72cd046/src/java/org/apache/cassandra/service/StorageService.java#L2548
> 
>
>
>
> Our nodes don’t run very hot, but it appears this streaming causes them to
> have issues. Have other people seen this ?
>
>
>
> Thanks !
>

-- 
This message may contain confidential and/or privileged information. 
If you are not the addressee or authorized to receive this on behalf of the 
addressee you must not use, copy, disclose or take action based on this 
message or any information herein. 
If you have received this message in error, please advise the sender 
immediately by reply email and delete this message. Thank you.

RE: RemoveNode CPU Spike Question

[Anubhav Kale]

Well, looking through logs I confirmed that my understanding below is correct, 
but would be good to hear from experts for sure 

From: Anubhav Kale [mailto:anubhav.k...@microsoft.com]
Sent: Tuesday, January 10, 2017 9:58 AM
To: user@cassandra.apache.org
Cc: Sean Usher 
Subject: RemoveNode CPU Spike Question

Hello,

Recently, I started noticing an interesting pattern. When I execute 
“removenode”, a subset of the nodes that now own the tokens result it in a CPU 
spike / disk activity, and sometimes SSTables on those nodes shoot up.

After looking through the code, it appears to me that below function forces 
data to be streamed from some of the new nodes to the node from where 
“removenode” is kicked in. Is my understanding correct ?

https://github.com/apache/cassandra/blob/d384e781d6f7c028dbe88cfe9dd3e966e72cd046/src/java/org/apache/cassandra/service/StorageService.java#L2548

Our nodes don’t run very hot, but it appears this streaming causes them to have 
issues. Have other people seen this ?

Thanks !