Hi all, We're seeing an issue where leader suffered a 3-second delay between elected to be the leader and listening for quorum connections.
------------------------------- 2025-03-27 00:44:44,872 INFO org.apache.zookeeper.server.quorum.QuorumPeer: Peer state changed: leading 2025-03-27 00:44:44,872 INFO org.apache.zookeeper.server.quorum.QuorumPeer: LEADING ... 2025-03-27 00:44:47,515 INFO org.apache.zookeeper.server.quorum.Leader: LEADING - LEADER ELECTION TOOK - 3078 MS 2025-03-27 00:44:47,515 INFO org.apache.zookeeper.server.quorum.QuorumPeer: Peer state changed: leading - discovery ------------------------------- We don't know what happened to the leader in these 3 secs, but it was more than enough for a single follower (other follower went down, hence the election) to give up connecting to leader. Got connection refused errors 5 times in a row. ------------------------------- 2025-03-27 00:44:45,269 ERROR org.apache.zookeeper.server.quorum.Learner: Unexpected exception, retries exceeded. tries=4 ------------------------------- After all this these 2 nodes had to wait for init limit to be expired (100 secs in this case, because tickTime=2000 and initLimit=50) for another round of leader election. Currently number of retries on follower side is hardcoded to 5 and leaderConnectDelayDuringRetryMs (not documented) defaults to 100ms, which means that this connection has to be finished in 0.5 secs otherwise follower will give up. Shall we change this somehow? - implement exponential backoff on follower side to wait longer, - make number of retries configurable, - increase default delay during retries to 1000ms (10x) Regards, Andor
