Re: Non-zero nodes are marked as down after restarting cassandra process

2017-05-16 Thread Jeff Jirsa
On 2017-05-16 07:07 (-0700), Andrew Jorgensen wrote: > Thanks for the info! > > When you say "overall stability problems due to some bugs", can you > elaborate on if those were bugs in cassandra that were fixed due to an > upgrade or bugs in your own code and how

Re: Non-zero nodes are marked as down after restarting cassandra process

2017-05-16 Thread Andrew Jorgensen
Thanks for the info! When you say "overall stability problems due to some bugs", can you elaborate on if those were bugs in cassandra that were fixed due to an upgrade or bugs in your own code and how you used cassandra. If the latter would it be possible to highlight what the most impactful fix

Re: Non-zero nodes are marked as down after restarting cassandra process

2017-03-01 Thread benjamin roth
You should always drain nodes before stopping the daemon whenever possible. This avoids commitlog replay on startup. This can take a while. But according to your description commit log replay seems not to be the cause. I once had a similar effect. Some nodes appeared down for some other nodes and

Re: Non-zero nodes are marked as down after restarting cassandra process

2017-03-01 Thread Ben Dalling
Hi Andrew, We were having problems with gossip TCP connections being held open and changed our SOP for stopping cassandra to being: nodetool disablegossip nodetool drain service cassandra stop This seemed to close down the gossip cleanly (the nodetool drain is advised as well) and meant that

Non-zero nodes are marked as down after restarting cassandra process

2017-03-01 Thread Andrew Jorgensen
Helllo, I have a cassandra cluster running on cassandra 3.0.3 and am seeing some strange behavior that I cannot explain when restarting cassandra nodes. The cluster is currently setup in a single datacenter and consists of 55 nodes. I am currently in the process of restarting nodes in the cluster