Also changing the compaction throughput on the fly while removing nodes is not scalable as we have 100s of nodes. I can try and test though
On Thursday, July 13, 2017, Jai Bheemsen Rao Dhanwada <jaibheem...@gmail.com> wrote: > Yes i did removenode and removenode force, and ecnountered same issue in > both the cases. > > On Thursday, July 13, 2017, Subroto Barua <sbarua...@yahoo.com.invalid> > wrote: > >> set streamthroughput higher than 200 on the source side and lower on the >> target node >> >> just curious, have you tried removenode force? >> >> >> On Thursday, July 13, 2017, 8:35:38 AM PDT, Jai Bheemsen Rao Dhanwada < >> jaibheem...@gmail.com> wrote: >> >> >> Thank you Sean, >> >> you mean setstreamthroughput to a lower value on the node where we are >> doing a "nodetool removenode <hostid>"? >> >> >> On Thu, Jul 13, 2017 at 8:07 AM, Durity, Sean R < >> sean_r_dur...@homedepot.com> wrote: >> >> Late to this party, but Jeff is talking about nodetool >> setstreamthroughput. The default in most versions is 200 Mb/s (set in yaml >> file as stream_throughput_outbound_ megabits_per_sec). This is outbound >> throttle only. So, if streams from multiple nodes are going to one, it can >> get inundated. >> >> >> >> The nodetool command lets you change this on the fly (no bounce >> required), but I don’t think it affects any current streaming from that >> node (only future). You can use nodetool getstreamthroughput to see the >> current value. >> >> >> >> >> >> Sean Durity >> >> >> >> *From:* Jai Bheemsen Rao Dhanwada [mailto:jaibheem...@gmail.com] >> *Sent:* Thursday, June 29, 2017 6:39 PM >> *To:* user@cassandra.apache.org >> *Subject:* Re: nodetool removenode causing the schema out of sync >> >> >> >> Thanks Jeff, >> >> >> >> Can you please suggest what value to tweak from the Cassandra side? >> >> >> >> On Thu, Jun 29, 2017 at 2:53 PM, Jeff Jirsa <jji...@apache.org> wrote: >> >> >> >> On 2017-06-29 13:45 (-0700), Jai Bheemsen Rao Dhanwada < >> jaibheem...@gmail.com> wrote: >> > Hello Jeff, >> > >> > Sorry the Version I am using 2.1.16, my first email had typo. >> > When I say schema out of sync >> > >> > 1. nodetool descriebcluster shows Schema versions same for all nodes. >> >> Ok got it, this is what I was most concerned with. >> >> > 2. nodetool removenode, shows the node down messages in the logs >> > 3. nodetool describecluster during this 1-2 mins shows several nodes as >> > UNREACHABLE and recovers with in a minute or two. >> >> This is likely due to overhead of streaming - you're probably running >> pretty close to your tipping point, and your streaming throughput creates >> enough GC pressure on the destinations to make them flap a bit. If you use >> the streaming throughput throttle, you may be able to help mitigate that >> somewhat (at the cost of speed). >> >> >> >> >> ------------------------------ ------------------------------ --------- >> To unsubscribe, e-mail: user-unsubscribe@cassandra. apache.org >> For additional commands, e-mail: user-h...@cassandra.apache.org >> >> >> >> ------------------------------ >> >> The information in this Internet Email is confidential and may be legally >> privileged. It is intended solely for the addressee. Access to this Email >> by anyone else is unauthorized. If you are not the intended recipient, any >> disclosure, copying, distribution or any action taken or omitted to be >> taken in reliance on it, is prohibited and may be unlawful. When addressed >> to our clients any opinions or advice contained in this Email are subject >> to the terms and conditions expressed in any applicable governing The Home >> Depot terms of business or client engagement letter. The Home Depot >> disclaims all responsibility and liability for the accuracy and content of >> this attachment and for any damages or losses arising from any >> inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other >> items of a destructive nature, which may be contained in this attachment >> and shall not be liable for direct, indirect, consequential or special >> damages in connection with this e-mail message or its attachment. >> >> >>
