[
https://issues.apache.org/jira/browse/ZOOKEEPER-2172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15418102#comment-15418102
]
Alexander Shraer commented on ZOOKEEPER-2172:
---------------------------------------------
Thanks Arshad, I verified that the test fails before the patch and passes
afterwards. I realize that the patch is complex, and appreciate you writing it!
I was trying to understand the logic.
A few minor comments:
- Please rename the test so that the name reflects what is being tested. Such
as ReconfigDuringLeaderSync / ReconfigDuringSnapshot or something similar.
- Please don't assume that SERVER_COUNT = 3 (so the extra server can be number
4). You could use a different variable in the test instead of SERVER_COUNT, or
just give the new server an id based on SERVER_COUNT instead of 4.
- The test shuts down the 4th server in one method and the other three in
another. Consider doing this in the same place and also closing the client
handles.
- getQP could be static, maybe rename to getCustomQuorumPeer or something like
that
- Please add more comments explaining the logic of the test. For example, in
writePacket please add a comment that you're delaying the ACK message a
follower sends as response to a NEWLEADER message, so that the leader has a
chance to send the reconfig and only then the UPTODATE (basically your comment
in this thread above). Without the comment its not clear why you're checking
for ACK while setting the newLeaderMessage flag. (I hope I understood the logic
correctly). Similarly, when the 4th server is being started, please add a
comment saying that this server will delay the response to a NEWLEADER message.
Basically, more comments would be helpful :)
- In the code, shouldn't the condition of the if be "pif.hdr.getZxid() ==
qp.getZxid() && qp.getType() == Leader.COMMITANDACTIVATE" ?
otherwise the logic inside the if may not be correct (the configuration info qv
is extracted from pif).
Maybe change to:
if (pif.hdr.getZxid() != qp.getZxid()) {
LOG.warn("Committing " + qp.getZxid() + ", but next
proposal is " + pif.hdr.getZxid());
} else if (qp.getType() == Leader.COMMITANDACTIVATE) {
> Cluster crashes when reconfig a new node as a participant
> ---------------------------------------------------------
>
> Key: ZOOKEEPER-2172
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2172
> Project: ZooKeeper
> Issue Type: Bug
> Components: leaderElection, quorum, server
> Affects Versions: 3.5.0
> Environment: Ubuntu 12.04 + java 7
> Reporter: Ziyou Wang
> Assignee: Arshad Mohammad
> Priority: Critical
> Fix For: 3.5.3
>
> Attachments: ZOOKEEPER-2172-02.patch, ZOOKEEPER-2172-03.patch,
> ZOOKEEPER-2172.patch, history.txt, node-1.log, node-2.log, node-3.log,
> zoo-1.log, zoo-2-1.log, zoo-2-2.log, zoo-2-3.log, zoo-2.log, zoo-2212-1.log,
> zoo-2212-2.log, zoo-2212-3.log, zoo-3-1.log, zoo-3-2.log, zoo-3-3.log,
> zoo-3.log, zoo-4-1.log, zoo-4-2.log, zoo-4-3.log, zoo.cfg.dynamic.10000005d,
> zoo.cfg.dynamic.next, zookeeper-1.log, zookeeper-1.out, zookeeper-2.log,
> zookeeper-2.out, zookeeper-3.log, zookeeper-3.out
>
>
> The operations are quite simple: start three zk servers one by one, then
> reconfig the cluster to add the new one as a participant. When I add the
> third one, the zk cluster may enter a weird state and cannot recover.
>
> I found “2015-04-20 12:53:48,236 [myid:1] - INFO [ProcessThread(sid:1
> cport:-1)::PrepRequestProcessor@547] - Incremental reconfig” in node-1 log.
> So the first node received the reconfig cmd at 12:53:48. Latter, it logged
> “2015-04-20 12:53:52,230 [myid:1] - ERROR
> [LearnerHandler-/10.0.0.2:55890:LearnerHandler@580] - Unexpected exception
> causing shutdown while sock still open” and “2015-04-20 12:53:52,231 [myid:1]
> - WARN [LearnerHandler-/10.0.0.2:55890:LearnerHandler@595] - ******* GOODBYE
> /10.0.0.2:55890 ********”. From then on, the first node and second node
> rejected all client connections and the third node didn’t join the cluster as
> a participant. The whole cluster was done.
>
> When the problem happened, all three nodes just used the same dynamic
> config file zoo.cfg.dynamic.10000005d which only contained the first two
> nodes. But there was another unused dynamic config file in node-1 directory
> zoo.cfg.dynamic.next which already contained three nodes.
>
> When I extended the waiting time between starting the third node and
> reconfiguring the cluster, the problem didn’t show again. So it should be a
> race condition problem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
