Vishal K updated ZOOKEEPER-822:

    Attachment: zookeeper-3.4.0.tar.gz


To rule out any setup specific issues (VM/ESX/ect), I tried to reproduce this 
problem with 3 physical machines (no VMs). I was able to reproduce the same 
problem after 15 reboots or so. It took a little more number of reboots on the 
physical setup. I used the latest code this time. I  have attached the logs. 

I am suspecting 3 problems:
1. blocking connect in WorkerSender.process as described in my earlier comments.
2. SendWorker.run() calls finish at the end. This could result in finish() 
getting called twice (e.g., finish called from receiveConnection), thus, 
causing  senderWorkerMap.remove(sid) called twice and removing an entry that 
should be removed.
3. Race condition that causes one of the peers to disconnect other peer's 
connection (receiveConnection/initiateConnection issue). I am still 
verifying/investigating the this.

In logs of server id 0 and 1, search for ***REBOOTING LEADER*** from the 
bottom. This line marks the start of faulty FLE.


> Leader election taking a long time  to complete
> -----------------------------------------------
>                 Key: ZOOKEEPER-822
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-822
>             Project: Zookeeper
>          Issue Type: Bug
>          Components: quorum
>    Affects Versions: 3.3.0
>            Reporter: Vishal K
>            Priority: Blocker
>         Attachments: 822.tar.gz, rhel.tar.gz, test_zookeeper_1.log, 
> test_zookeeper_2.log, zk_leader_election.tar.gz, zookeeper-3.4.0.tar.gz
> Created a 3 node cluster.
> 1 Fail the ZK leader
> 2. Let leader election finish. Restart the leader and let it join the 
> 3. Repeat 
> After a few rounds leader election takes anywhere 25- 60 seconds to finish. 
> Note- we didn't have any ZK clients and no new znodes were created.
> zoo.cfg is shown below:
> #Mon Jul 19 12:15:10 UTC 2010
> server.1=\:2888\:3888
> server.0=\:2888\:3888
> clientPort=2181
> dataDir=/var/zookeeper
> syncLimit=2
> server.2=\:2888\:3888
> initLimit=5
> tickTime=2000
> I have attached logs from two nodes that took a long time to form the cluster 
> after failing the leader. The leader was down anyways so logs from that node 
> shouldn't matter.
> Look for "START HERE". Logs after that point should be of our interest.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

Reply via email to