zhanglu153 created ZOOKEEPER-4781:
-
Summary: ZooKeeper not starting because the accepted epoch is less
than the current epoch.
Key: ZOOKEEPER-4781
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-4781
Project: ZooKeeper
Issue Type: Bug
Components: server
Affects Versions: 3.9.1, 3.8.3, 3.7.2, 3.6.4, 3.4.14, 3.5.10
Reporter: zhanglu153
This issue occurred in our abnormal testing environment, where the disk was
injected with anomalies and frequently filled up.
The the scenario is as follows:
# Configure three node ZooKeeper cluster, lets say nodes are A, B and C.
# Start the cluster, and node C becomes the leader.
# The disk of Node C is injected with a full disk exception.
# Node C called the org.apache.zookeeper.server.quorum.Leader#lead method.
{code:java}
void lead() throws IOException, InterruptedException {
self.end_fle = Time.currentElapsedTime();
long electionTimeTaken = self.end_fle - self.start_fle;
self.setElectionTimeTaken(electionTimeTaken);
LOG.info("LEADING - LEADER ELECTION TOOK - {}", electionTimeTaken);
self.start_fle = 0;
self.end_fle = 0;
zk.registerJMX(new LeaderBean(this, zk), self.jmxLocalPeerBean);
try {
self.tick.set(0);
zk.loadData();
leaderStateSummary = new StateSummary(self.getCurrentEpoch(),
zk.getLastProcessedZxid());
// Start thread that waits for connection requests from
// new followers.
cnxAcceptor = new LearnerCnxAcceptor();
cnxAcceptor.start();
readyToStart = true;
long epoch = getEpochToPropose(self.getId(), self.getAcceptedEpoch());
zk.setZxid(ZxidUtils.makeZxid(epoch, 0));
synchronized(this){
lastProposed = zk.getZxid();
}
newLeaderProposal.packet = new QuorumPacket(NEWLEADER, zk.getZxid(),
null, null);
if ((newLeaderProposal.packet.getZxid() & 0xL) != 0) {
LOG.info("NEWLEADER proposal has Zxid of "
+ Long.toHexString(newLeaderProposal.packet.getZxid()));
}
waitForEpochAck(self.getId(), leaderStateSummary);
self.setCurrentEpoch(epoch);
... {code}
# Node C, as the leader, will start the LearnerCnxAcceptor thread, and then
call the org.apache.zookeeper.server.quorum.Leader#getEpochToPropose method. At
this time, the value of waitingForNewEpoch is true, and the size of
connectingFollowers is not greater than n/2. Node C directly calls
connectingFollowers.wait to wait. The maximum waiting time is
self.getInitLimit()*self.getTickTime() ms.
{code:java}
public long getEpochToPropose(long sid, long lastAcceptedEpoch) throws
InterruptedException, IOException {
synchronized(connectingFollowers) {
if (!waitingForNewEpoch) {
return epoch;
}
if (lastAcceptedEpoch >= epoch) {
epoch = lastAcceptedEpoch+1;
}
if (isParticipant(sid)) {
connectingFollowers.add(sid);
}
QuorumVerifier verifier = self.getQuorumVerifier();
if (connectingFollowers.contains(self.getId()) &&
verifier.containsQuorum(connectingFollowers)) {
self.setAcceptedEpoch(epoch);
waitingForNewEpoch = false;
connectingFollowers.notifyAll();
} else {
long start = Time.currentElapsedTime();
long cur = start;
long end = start + self.getInitLimit()*self.getTickTime();
while(waitingForNewEpoch && cur < end) {
connectingFollowers.wait(end - cur);
cur = Time.currentElapsedTime();
}
if (waitingForNewEpoch) {
throw new InterruptedException("Timeout while waiting for epoch
from quorum");
}
}
return epoch;
}
} {code}
# Node B connects to the 2888 communication port of node C and starts a new
LeanerHandler thread.
# Node A connects to the 2888 communication port of node C and starts a new
LeanerHandler thread.
# After node B connects to node C, call the
org.apache.zookeeper.server.quorum.Leader#getEpochToPropose method in the
LearnerHandler thread.At this point, the value of waitingForNewEpoch is true,
and the size of connectingFollowers is greater than n/2. Then, set the value of
waitingForNewEpoch to false. Due to the disk of node C being full, calling
setAcceptedEpoch to write the acceptedEpoch value failed with an IO exception.
Node C fails to update the acceptedEpoch file and did not successfully call the
connectingFollowers.notifyAll() method. This will cause node C to wait at
connectingFollowers.wait, with a maximum wait of
self.getInitLimit()*self.getTickTime() ms. Due to an IO exception thrown, the
socket connection
