Alan Horn created ZOOKEEPER-1548:
------------------------------------
Summary: Cluster fails election loop in new and interesting way
Key: ZOOKEEPER-1548
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1548
Project: ZooKeeper
Issue Type: Bug
Components: leaderElection
Affects Versions: 3.4.3
Reporter: Alan Horn
Hi,
We have a five node cluster, recently upgraded from 3.3.5 to 3.4.3. Was running
fine for a few weeks after the upgrade, then the following sequence of events
occurred :
1. All servers stopped responding to 'ruok' at the same time
2. Our local supervisor process restarted all of them at the same time
(yes, this is bad, we didn't expect it to fail this way :)
3. The cluster would not serve requests after this. Appeared to be unable to
complete an election.
We tried various things at this point, none of which worked :
* Moved around the restart order of the nodes (e.g. 4 thru 0, instead of 0 thru
4)
* Reduced number of running nodes from 5 -> 3 to simplify the quorum, by only
starting up 0, 1 & 2, in one test, and 0, 2 & 4 in the other
* Removed the *Epoch files from version-2/ snapshot directory
* Put the same version2/snapshot.xxxxx file on each server in the cluster
* Added the (same on all nodes) last txlog onto each cluster
* Kept only the last snapshot plus txlog unique on each server
* Moved leaderServes=no to leaderServes=yes
* Removed all files and started up with empty data as a control. This worked,
but of course isn't terribly useful :)
Finally, I brought the data up on a single node running in standalone and this
worked (yay!) So at this point we brought the single node back into service and
have kept the other four available to debug why the election is failing.
We downgraded the four nodes to 3.3.5, and then they completed the election and
started serving as expected.
We did a rolling upgrade to 3.4.3, and everything was fine until we restarted
the leader, whereupon we encountered the same re-election loop as before.
We're a bit out of ideas at this point, so I was hoping someone from this list
might have some useful input.
Output from two followers and a leader during this condition are attached.
Cheers,
Al
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
