set streamthroughput higher than 200 on the source side and lower on the target 
just curious, have you tried removenode force?

On Thursday, July 13, 2017, 8:35:38 AM PDT, Jai Bheemsen Rao Dhanwada 
<> wrote:

Thank you Sean,
you mean setstreamthroughput to a lower value on the node where we are doing a 
"nodetool removenode <hostid>"?

On Thu, Jul 13, 2017 at 8:07 AM, Durity, Sean R <> 

Late to this party, but Jeff is talking about nodetool setstreamthroughput. The 
default in most versions is 200 Mb/s (set in yaml file as 
stream_throughput_outbound_ megabits_per_sec). This is outbound throttle only. 
So, if streams from multiple nodes are going to one, it can get inundated.


The nodetool command lets you change this on the fly (no bounce required), but 
I don’t think it affects any current streaming from that node (only future). 
You can use nodetool getstreamthroughput to see the current value.



Sean Durity


From: Jai Bheemsen Rao Dhanwada []
Sent: Thursday, June 29, 2017 6:39 PM
Subject: Re: nodetool removenode causing the schema out of sync


Thanks Jeff,


Can you please suggest what value to tweak from the Cassandra side?


On Thu, Jun 29, 2017 at 2:53 PM, Jeff Jirsa <> wrote:

On 2017-06-29 13:45 (-0700), Jai Bheemsen Rao Dhanwada <> 
> Hello Jeff,
> Sorry the Version I am using 2.1.16, my first email had typo.
> When I say schema out of sync
> 1. nodetool descriebcluster shows Schema versions same for all nodes.

Ok got it, this is what I was most concerned with.

> 2. nodetool removenode, shows the node down messages in the logs
> 3. nodetool describecluster during this 1-2 mins shows several nodes as
> UNREACHABLE and recovers with in a minute or two.

This is likely due to overhead of streaming - you're probably running pretty 
close to your tipping point, and your streaming throughput creates enough GC 
pressure on the destinations to make them flap a bit. If you use the streaming 
throughput throttle, you may be able to help mitigate that somewhat (at the 
cost of speed).

------------------------------ ------------------------------ ---------
To unsubscribe, e-mail: user-unsubscribe@cassandra.
For additional commands, e-mail:


The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

Reply via email to