[
https://issues.apache.org/jira/browse/ZOOKEEPER-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matan updated ZOOKEEPER-2081:
-----------------------------
Attachment: zk-3.log
zk-1.log
It's been a while so I tried this again on ZK 3.4.6 following the same steps I
outlined above and it is consistent. DEBUG logs are attached.
Note that I used 88.198.26.2 as the dummy IP for this test that I blackholed.
I'm interested to know if this is by design or actually an issue.
Thanks!
> Leader election cannot complete when a node is blackholed (unreachable) even
> when quorum is possible.
> -----------------------------------------------------------------------------------------------------
>
> Key: ZOOKEEPER-2081
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2081
> Project: ZooKeeper
> Issue Type: Bug
> Components: leaderElection, quorum
> Affects Versions: 3.3.6, 3.4.6
> Environment: Verified on RHEL and Mac OS X.
> Reporter: Matan
> Attachments: zk-1.log, zk-3.log
>
>
> I noticed a situation when one of our 3-node clusters on RHEL lost a machine
> due to PSU failure. The remaining two nodes failed to complete leader
> election and would continually restart the leader election process.
> Restarting the nodes would not help and they would reach the same exact state.
> This was curious so I spent some time and managed to reproduce this on my
> local machine and found what looks like the main factor:
> When a node is unreachable (timeouts), this somehow causes the election
> process to get out of sync. Once a leader is decided, the follower tries to
> connect to the leader only when the leader is not listening.
> Then the follower gives up and the process starts again ad infinitum.
> How to reproduce on a local machine:
> 1. Setup up a 3 node cluster of ZK. Note we only need to set up 2 boxes
> since we'll just make the third unreachable:
> MyId 1:
> server.1=MyMachine:2881:3881
> server.2=<Put any IP that we can block>:2882:3882
> server.3=MyMachine:2883:3883
> MyId 3:
> server.1=MyMachine:2881:3881
> server.2=<Put any IP that we can block>:2882:3882
> server.3=MyMachine:2883:3883
> Now set up a blackhole route for the IP you choose (Mac OSX, Linux is
> similar):
> > route add -host <IP you selected> 127.0.0.1 -blackhole
> Start your 2 nodes. They will never reach quorum.
> However, if I remove the blackhole route and just not start the 3rd instance
> (but the host is still reachable), it will work fine and quorum will be
> reached almost immediately.
> It seems the difference between the “timeout” and a "connection refused”
> makes all the difference somehow in the election process.
> I verified this behavior on 3.4.6 and 3.3.6.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
