Answers inline

Jeff Jirsa

> On Aug 12, 2017, at 2:58 PM, wrote:
> Hi folks, hopefully a quick one:
> We are running a 12 node cluster (2.1.15) in AWS with Ec2Snitch.  It's all in 
> one region but spread across 3 availability zones.  It was nicely balanced 
> with 4 nodes in each.
> But with a couple of failures and subsequent provisions to the wrong az we 
> now have a cluster with : 
> 5 nodes in az A
> 5 nodes in az B
> 2 nodes in az C
> Not sure why, but when adding a third node in AZ C it fails to stream after 
> getting all the way to completion and no apparent error in logs.  I've looked 
> at a couple of bugs referring to scrubbing and possible OOM bugs due to 
> metadata writing at end of streaming (sorry don't have ticket handy).  I'm 
> worried I might not be able to do much with these since the disk space usage 
> is high and they are under a lot of load given the small number of them for 
> this rack.

You'll definitely have higher load on az C instances with rf=3 in this ratio

Streaming should still work - are you sure it's not busy doing something? Like 
building secondary index or similar? jstack thread dump would be useful, or at 
least nodetool tpstats

> Rather than troubleshoot this further, what I was thinking about doing was:
> - drop the replication factor on our keyspace to two

Repair before you do this, or you'll lose your consistency guarantees

> - hopefully this would reduce load on these two remaining nodes 

It should, racks awareness guarantees on replica per rack if rf==num racks, so 
right now those 2 c machines have 2.5x as much data as the others. This will 
drop that requirement and drop the load significantly 

> - run repairs/cleanup across the cluster 
> - then shoot these two nodes in the 'c' rack

Why shoot the c instances? Why not drop RF and then add 2 more C instances, 
then increase RF back to 3, run repair, then Decom the extra instances in a and 

> - run repairs/cleanup across the cluster
> Would this work with minimal/no disruption? 

The big risk of running rf=2 is that quorum==all - any gc pause or node 
restarting will make you lose HA or strong consistency guarantees.

> Should I update their "rack" before hand or after ?

You can't change a node's rack once it's in the cluster, it SHOULD refuse to 
start if you do that

> What else am I not thinking about? 
> My main goal atm is to get back to where the cluster is in a clean consistent 
> state that allows nodes to properly bootstrap.
> Thanks for your help in advance.
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

Reply via email to