Hi Anubhav, This happened to us as well, on all nodes in the DC. We found that after performing removenode, all other nodes suddenly started to do a lot of compactions that increased CPU. To mitigate that, we used nodetool disableautocompaction before removing the node. Then, after removal, we slowly enabled autocompaction (a few minutes between each enable) on the nodes one by one. This helped with the CPU increase you've mentioned.
Shalom Sagges DBA T: +972-74-700-4035 <http://www.linkedin.com/company/164748> <http://twitter.com/liveperson> <http://www.facebook.com/LivePersonInc> We Create Meaningful Connections <https://liveperson.docsend.com/view/8iiswfp> On Tue, Jan 10, 2017 at 8:03 PM, Anubhav Kale <anubhav.k...@microsoft.com> wrote: > Well, looking through logs I confirmed that my understanding below is > correct, but would be good to hear from experts for sure š > > > > *From:* Anubhav Kale [mailto:anubhav.k...@microsoft.com] > *Sent:* Tuesday, January 10, 2017 9:58 AM > *To:* user@cassandra.apache.org > *Cc:* Sean Usher <seus...@exchange.microsoft.com> > *Subject:* RemoveNode CPU Spike Question > > > > Hello, > > > > Recently, I started noticing an interesting pattern. When I execute > āremovenodeā, a subset of the nodes that now own the tokens result it in a > CPU spike / disk activity, and sometimes SSTables on those nodes shoot up. > > > > After looking through the code, it appears to me that below function > forces data to be streamed from some of the new nodes to the node from > where āremovenodeā is kicked in. Is my understanding correct ? > > > > https://github.com/apache/cassandra/blob/d384e781d6f7c028dbe88cfe9dd3e9 > 66e72cd046/src/java/org/apache/cassandra/service/StorageService.java#L2548 > <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fcassandra%2Fblob%2Fd384e781d6f7c028dbe88cfe9dd3e966e72cd046%2Fsrc%2Fjava%2Forg%2Fapache%2Fcassandra%2Fservice%2FStorageService.java%23L2548&data=02%7C01%7CAnubhav.Kale%40microsoft.com%7C173daa48fcaf4ca6498d08d43982318c%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636196678720784947&sdata=JZ9zWh%2FtJJ%2FbhXXkT41yQhANKaUSBHfP53WraY2vL8M%3D&reserved=0> > > > > Our nodes donāt run very hot, but it appears this streaming causes them to > have issues. Have other people seen this ? > > > > Thanks ! > -- This message may contain confidential and/or privileged information. If you are not the addressee or authorized to receive this on behalf of the addressee you must not use, copy, disclose or take action based on this message or any information herein. If you have received this message in error, please advise the sender immediately by reply email and delete this message. Thank you.
