Philipp created ZOOKEEPER-3776:
----------------------------------
Summary: Cluster stuck not forming up quorum
Key: ZOOKEEPER-3776
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3776
Project: ZooKeeper
Issue Type: Bug
Reporter: Philipp
We had working Zookeeper 3.4.8 that came with Kafka. 3 machines cluster. Now
I'm trying to upgrade it to 3.6.0. I didn't touch the data directory, just set
up 3.6.0 Zookeeper in the new directory and updated config. Stopped all
Zookeepers. Now I've started eache Zk in turn with some delay. And they are not
forming up quorum, stuck seemingly forever, sometimes trying to elect and
sometimes throwing nonsensical errors.
For example, on node 1 (sorry don't know how to format this properly in jira):
Mar 31 20:05:21 java[24055]: 2020-03-31 20:05:21,133 [myid:1] - INFO
[WorkerReceiver[myid=1]:FastLeaderElection$Messenger$WorkerReceiver@376] -
Notification: my state:LOOKING; n.sid:2, n.state:LOOKING, n.leader:2,
n.round:0x4b, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2,
n.config version:0x0
Mar 31 20:05:21 java[24055]: 2020-03-31 20:05:21,133 [myid:1] - INFO
[WorkerReceiver[myid=1]:FastLeaderElection$Messenger$WorkerReceiver@376] -
Notification: my state:LOOKING; n.sid:1, n.state:LOOKING, n.leader:2,
n.round:0x4b, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2,
n.config version:0x0
Mar 31 20:05:21 java[24055]: 2020-03-31 20:05:21,134 [myid:1] - INFO
[WorkerReceiver[myid=1]:FastLeaderElection$Messenger$WorkerReceiver@376] -
Notification: my state:LOOKING; n.sid:3, n.state:LEADING, n.leader:3,
n.round:0x4a, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2,
n.config version:0x0
Mar 31 20:05:21 java[24055]: 2020-03-31 20:05:21,335 [myid:1] - INFO
[WorkerReceiver[myid=1]:FastLeaderElection$Messenger$WorkerReceiver@376] -
Notification: my state:LOOKING; n.sid:2, n.state:LOOKING, n.leader:2,
n.round:0x4b, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2,
n.config version:0x0
Mar 31 20:05:21 java[24055]: 2020-03-31 20:05:21,536 [myid:1] - INFO
[QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):QuorumPeer@857] - Peer
state changed: following
Mar 31 20:05:21 java[24055]: 2020-03-31 20:05:21,536 [myid:1] - INFO
[QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):QuorumPeer@1453] -
FOLLOWING
Mar 31 20:05:21 java[24055]: 2020-03-31 20:05:21,536 [myid:1] - INFO
[QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):ZooKeeperServer@1246]
- minSessionTimeout set to 6000
Mar 31 20:05:21 java[24055]: 2020-03-31 20:05:21,536 [myid:1] - INFO
[QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):ZooKeeperServer@1255]
- maxSessionTimeout set to 60000
Mar 31 20:05:21 java[24055]: 2020-03-31 20:05:21,536 [myid:1] - INFO
[QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):ResponseCache@45] -
Response cache size is initialized with value 400.
Mar 31 20:05:21 java[24055]: 2020-03-31 20:05:21,536 [myid:1] - INFO
[QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):RequestPathMetricsCollector@111]
- zookeeper.pathStats.slotCapacity = 60
Mar 31 20:05:21 java[24055]: 2020-03-31 20:05:21,536 [myid:1] - INFO
[QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):RequestPathMetricsCollector@112]
- zookeeper.pathStats.slotDuration = 15
Mar 31 20:05:21 java[24055]: 2020-03-31 20:05:21,536 [myid:1] - INFO
[QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):RequestPathMetricsCollector@113]
- zookeeper.pathStats.maxDepth = 6
Mar 31 20:05:21 java[24055]: 2020-03-31 20:05:21,536 [myid:1] - INFO
[QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):RequestPathMetricsCollector@114]
- zookeeper.pathStats.initialDelay = 5
Mar 31 20:05:21 java[24055]: 2020-03-31 20:05:21,537 [myid:1] - INFO
[QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):RequestPathMetricsCollector@115]
- zookeeper.pathStats.delay = 5
Mar 31 20:05:21 java[24055]: 2020-03-31 20:05:21,537 [myid:1] - INFO
[QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):RequestPathMetricsCollector@116]
- zookeeper.pathStats.enabled = false
Mar 31 20:05:21 java[24055]: 2020-03-31 20:05:21,537 [myid:1] - INFO
[QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):ZooKeeperServer@1470]
- The max bytes for all large requests are set to 104857600
Mar 31 20:05:21 java[24055]: 2020-03-31 20:05:21,537 [myid:1] - INFO
[QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):ZooKeeperServer@1484]
- The large request threshold is set to -1
Mar 31 20:05:21 java[24055]: 2020-03-31 20:05:21,537 [myid:1] - INFO
[QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):ZooKeeperServer@329] -
Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout
60000 clientPortListenBacklog -1 datadir
/kafka/kafka_zookeeper_data/logdir/version-2 snapdir
/kafka/kafka_zookeeper_data/version-2
Mar 31 20:05:21 java[24055]: 2020-03-31 20:05:21,537 [myid:1] - INFO
[QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):Follower@75] -
FOLLOWING - LEADER ELECTION TOOK - 92790 MS
Mar 31 20:05:21 java[24055]: 2020-03-31 20:05:21,537 [myid:1] - INFO
[QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):QuorumPeer@863] - Peer
state changed: following - discovery
Mar 31 20:05:21 java[24055]: 2020-03-31 20:05:21,736 [myid:1] - INFO
[WorkerReceiver[myid=1]:FastLeaderElection$Messenger$WorkerReceiver@376] -
Notification: my state:FOLLOWING; n.sid:2, n.state:LOOKING, n.leader:2,
n.round:0x4b, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2,
n.config version:0x0
Mar 31 20:05:22 java[24055]: 2020-03-31 20:05:22,538 [myid:1] - INFO
[WorkerReceiver[myid=1]:FastLeaderElection$Messenger$WorkerReceiver@376] -
Notification: my state:FOLLOWING; n.sid:2, n.state:LOOKING, n.leader:2,
n.round:0x4b, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2,
n.config version:0x0
Mar 31 20:05:22 java[24055]: 2020-03-31 20:05:22,539 [myid:1] - WARN
[QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):Follower@129] -
Exception when following the leader
Mar 31 20:05:22 java[24055]: java.lang.IllegalArgumentException
Mar 31 20:05:22 java[24055]: at
java.util.concurrent.ThreadPoolExecutor.<init>(ThreadPoolExecutor.java:1307)
Mar 31 20:05:22 java[24055]: at
java.util.concurrent.ThreadPoolExecutor.<init>(ThreadPoolExecutor.java:1195)
Mar 31 20:05:22 java[24055]: at
java.util.concurrent.Executors.newFixedThreadPool(Executors.java:89)
Mar 31 20:05:22 java[24055]: at
org.apache.zookeeper.server.quorum.Learner.connectToLeader(Learner.java:275)
Mar 31 20:05:22 java[24055]: at
org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:87)
Mar 31 20:05:22 java[24055]: at
org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1455)
Mar 31 20:05:22 java[24055]: 2020-03-31 20:05:22,539 [myid:1] - INFO
[QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):Follower@292] -
shutdown Follower
Mar 31 20:05:22 java[24055]: 2020-03-31 20:05:22,539 [myid:1] - INFO
[QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):QuorumPeer@863] - Peer
state changed: looking
Mar 31 20:05:22 java[24055]: 2020-03-31 20:05:22,539 [myid:1] - WARN
[QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):QuorumPeer@1501] -
PeerState set to LOOKING
Mar 31 20:05:22 java[24055]: 2020-03-31 20:05:22,539 [myid:1] - INFO
[QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):QuorumPeer@1371] -
LOOKING
Mar 31 20:05:22 java[24055]: 2020-03-31 20:05:22,539 [myid:1] - INFO
[QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):FastLeaderElection@931]
- New election. My id = 1, proposed zxid=0x10002b653
Mar 31 20:05:22 java[24055]: 2020-03-31 20:05:22,539 [myid:1] - INFO
[WorkerReceiver[myid=1]:FastLeaderElection$Messenger$WorkerReceiver@376] -
Notification: my state:LOOKING; n.sid:1, n.state:LOOKING, n.leader:1,
n.round:0x4c, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2,
n.config version:0x0
Mar 31 20:05:22 java[24055]: 2020-03-31 20:05:22,542 [myid:1] - INFO
[WorkerReceiver[myid=1]:FastLeaderElection$Messenger$WorkerReceiver@376] -
Notification: my state:LOOKING; n.sid:3, n.state:LEADING, n.leader:3,
n.round:0x4a, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2,
n.config version:0x0
Mar 31 20:05:22 java[24055]: 2020-03-31 20:05:22,544 [myid:1] - INFO
[WorkerReceiver[myid=1]:FastLeaderElection$Messenger$WorkerReceiver@376] -
Notification: my state:LOOKING; n.sid:2, n.state:LEADING, n.leader:2,
n.round:0x4b, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2,
n.config version:0x0
Mar 31 20:05:22 java[24055]: 2020-03-31 20:05:22,744 [myid:1] - INFO
[WorkerReceiver[myid=1]:FastLeaderElection$Messenger$WorkerReceiver@376] -
Notification: my state:LOOKING; n.sid:1, n.state:LOOKING, n.leader:1,
n.round:0x4c, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2,
n.config version:0x0
Mar 31 20:05:22 java[24055]: 2020-03-31 20:05:22,744 [myid:1] - INFO
[QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):FastLeaderElection@966]
- Notification time out: 400
Mar 31 20:05:22 java[24055]: 2020-03-31 20:05:22,745 [myid:1] - INFO
[WorkerReceiver[myid=1]:FastLeaderElection$Messenger$WorkerReceiver@376] -
Notification: my state:LOOKING; n.sid:2, n.state:LEADING, n.leader:2,
n.round:0x4b, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2,
n.config version:0x0
Mar 31 20:05:22 java[24055]: 2020-03-31 20:05:22,746 [myid:1] - INFO
[WorkerReceiver[myid=1]:FastLeaderElection$Messenger$WorkerReceiver@376] -
Notification: my state:LOOKING; n.sid:3, n.state:LEADING, n.leader:3,
n.round:0x4a, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2,
n.config version:0x0{code}
On node 2 in the same time:
Mar 31 20:04:39 java[23556]: 2020-03-31 20:04:39,764 [myid:2] - INFO
[WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@376] -
Notification: my state:LEADING; n.sid:1, n.state:LOOKING, n.leader:1,
n.round:0x4b, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2,
n.config version:0x0
Mar 31 20:05:21 java[23556]: 2020-03-31 20:05:21,130 [myid:2] - WARN
[QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):QuorumPeer@1471] -
Unexpected exception
Mar 31 20:05:21 java[23556]: java.lang.InterruptedException: Timeout while
waiting for epoch from quorum
Mar 31 20:05:21 java[23556]: at
org.apache.zookeeper.server.quorum.Leader.getEpochToPropose(Leader.java:1425)
Mar 31 20:05:21 java[23556]: at
org.apache.zookeeper.server.quorum.Leader.lead(Leader.java:599)
Mar 31 20:05:21 java[23556]: at
org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1468)
Mar 31 20:05:21 java[23556]: 2020-03-31 20:05:21,131 [myid:2] - INFO
[QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):Leader@792] - Shutting
down
Mar 31 20:05:21 java[23556]: 2020-03-31 20:05:21,131 [myid:2] - INFO
[QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):Leader@798] - Shutdown
called. For the reason Forcing shutdown
Mar 31 20:05:21 java[23556]: 2020-03-31 20:05:21,131 [myid:2] - INFO
[QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):QuorumPeer@863] - Peer
state changed: looking
Mar 31 20:05:21 java[23556]: 2020-03-31 20:05:21,131 [myid:2] - WARN
[QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):QuorumPeer@1501] -
PeerState set to LOOKING
Mar 31 20:05:21 java[23556]: 2020-03-31 20:05:21,131 [myid:2] - INFO
[QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):QuorumPeer@1371] -
LOOKING
Mar 31 20:05:21 java[23556]: 2020-03-31 20:05:21,131 [myid:2] - WARN
[LearnerCnxAcceptorHandler-/172.30.2.252:2888:Leader$LearnerCnxAcceptor$LearnerCnxAcceptorHandler@523]
- Exception while shutting down acceptor.
Mar 31 20:05:21 java[23556]: java.net.SocketException: Socket closed
Mar 31 20:05:21 java[23556]: at
java.net.PlainSocketImpl.socketAccept(Native Method)
Mar 31 20:05:21 java[23556]: at
java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
Mar 31 20:05:21 java[23556]: at
java.net.ServerSocket.implAccept(ServerSocket.java:545)
Mar 31 20:05:21 java[23556]: at
java.net.ServerSocket.accept(ServerSocket.java:513)
Mar 31 20:05:21 java[23556]: at
org.apache.zookeeper.server.quorum.Leader$LearnerCnxAcceptor$LearnerCnxAcceptorHandler.acceptConnections(Leader.java:510)
Mar 31 20:05:21 java[23556]: at
org.apache.zookeeper.server.quorum.Leader$LearnerCnxAcceptor$LearnerCnxAcceptorHandler.run(Leader.java:493)
Mar 31 20:05:21 java[23556]: at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
Mar 31 20:05:21 java[23556]: at
java.util.concurrent.FutureTask.run(FutureTask.java:266)
Mar 31 20:05:21 java[23556]: at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
Mar 31 20:05:21 java[23556]: at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Mar 31 20:05:21 java[23556]: at java.lang.Thread.run(Thread.java:745)
Mar 31 20:05:21 java[23556]: 2020-03-31 20:05:21,131 [myid:2] - INFO
[QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):FastLeaderElection@931]
- New election. My id = 2, proposed zxid=0x10002b653
Mar 31 20:05:21 java[23556]: 2020-03-31 20:05:21,133 [myid:2] - INFO
[WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@376] -
Notification: my state:LOOKING; n.sid:2, n.state:LOOKING, n.leader:2,
n.round:0x4b, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2,
n.config version:0x0
Mar 31 20:05:21 java[23556]: 2020-03-31 20:05:21,134 [myid:2] - INFO
[WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@376] -
Notification: my state:LOOKING; n.sid:1, n.state:LOOKING, n.leader:2,
n.round:0x4b, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2,
n.config version:0x0
Mar 31 20:05:21 java[23556]: 2020-03-31 20:05:21,134 [myid:2] - INFO
[WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@376] -
Notification: my state:LOOKING; n.sid:3, n.state:LEADING, n.leader:3,
n.round:0x4a, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2,
n.config version:0x0
Mar 31 20:05:21 java[23556]: 2020-03-31 20:05:21,335 [myid:2] - INFO
[QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):FastLeaderElection@966]
- Notification time out: 400
Mar 31 20:05:21 java[23556]: 2020-03-31 20:05:21,335 [myid:2] - INFO
[WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@376] -
Notification: my state:LOOKING; n.sid:2, n.state:LOOKING, n.leader:2,
n.round:0x4b, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2,
n.config version:0x0
Mar 31 20:05:21 java[23556]: 2020-03-31 20:05:21,336 [myid:2] - INFO
[WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@376] -
Notification: my state:LOOKING; n.sid:3, n.state:LEADING, n.leader:3,
n.round:0x4a, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2,
n.config version:0x0
Mar 31 20:05:21 java[23556]: 2020-03-31 20:05:21,736 [myid:2] - INFO
[QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):FastLeaderElection@966]
- Notification time out: 800
Mar 31 20:05:21 java[23556]: 2020-03-31 20:05:21,736 [myid:2] - INFO
[WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@376] -
Notification: my state:LOOKING; n.sid:2, n.state:LOOKING, n.leader:2,
n.round:0x4b, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2,
n.config version:0x0
Mar 31 20:05:21 java[23556]: 2020-03-31 20:05:21,737 [myid:2] - INFO
[WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@376] -
Notification: my state:LOOKING; n.sid:1, n.state:FOLLOWING, n.leader:2,
n.round:0x4b, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2,
n.config version:0x0
Mar 31 20:05:21 java[23556]: 2020-03-31 20:05:21,737 [myid:2] - INFO
[WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@376] -
Notification: my state:LOOKING; n.sid:3, n.state:LEADING, n.leader:3,
n.round:0x4a, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2,
n.config version:0x0
Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,538 [myid:2] - INFO
[QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):FastLeaderElection@966]
- Notification time out: 1600
Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,538 [myid:2] - INFO
[WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@376] -
Notification: my state:LOOKING; n.sid:1, n.state:FOLLOWING, n.leader:2,
n.round:0x4b, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2,
n.config version:0x0
Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,539 [myid:2] - INFO
[QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):QuorumPeer@857] - Peer
state changed: leading
Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,539 [myid:2] - INFO
[QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):QuorumPeer@1465] -
LEADING
Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,539 [myid:2] - INFO
[QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):ZooKeeperServer@1246]
- minSessionTimeout set to 6000
Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,539 [myid:2] - INFO
[QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):ZooKeeperServer@1255]
- maxSessionTimeout set to 60000
Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,539 [myid:2] - INFO
[QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):ResponseCache@45] -
Response cache size is initialized with value 400.
Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,539 [myid:2] - INFO
[QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):RequestPathMetricsCollector@111]
- zookeeper.pathStats.slotCapacity = 60
Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,539 [myid:2] - INFO
[QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):RequestPathMetricsCollector@112]
- zookeeper.pathStats.slotDuration = 15
Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,539 [myid:2] - INFO
[QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):RequestPathMetricsCollector@113]
- zookeeper.pathStats.maxDepth = 6
Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,540 [myid:2] - INFO
[WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@376] -
Notification: my state:LEADING; n.sid:1, n.state:LOOKING, n.leader:1,
n.round:0x4c, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2,
n.config version:0x0
Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,540 [myid:2] - INFO
[QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):RequestPathMetricsCollector@114]
- zookeeper.pathStats.initialDelay = 5
Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,541 [myid:2] - INFO
[QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):RequestPathMetricsCollector@115]
- zookeeper.pathStats.delay = 5
Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,541 [myid:2] - INFO
[QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):RequestPathMetricsCollector@116]
- zookeeper.pathStats.enabled = false
Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,541 [myid:2] - INFO
[QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):ZooKeeperServer@1470]
- The max bytes for all large requests are set to 104857600
Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,541 [myid:2] - INFO
[QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):ZooKeeperServer@1484]
- The large request threshold is set to -1
Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,541 [myid:2] - INFO
[QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):ZooKeeperServer@329] -
Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout
60000 clientPortListenBacklog -1 datadir
/kafka/kafka_zookeeper_data/logdir/version-2 snapdir
/kafka/kafka_zookeeper_data/version-2
Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,541 [myid:2] - INFO
[QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):Leader@581] - LEADING
- LEADER ELECTION TOOK - 1410 MS
Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,541 [myid:2] - INFO
[QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):QuorumPeer@863] - Peer
state changed: leading - discovery
Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,541 [myid:2] - INFO
[QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):FileTxnSnapLog@470] -
Snapshotting: 0x10002b653 to
/kafka/kafka_zookeeper_data/version-2/snapshot.10002b653
Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,544 [myid:2] - INFO
[WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@376] -
Notification: my state:LEADING; n.sid:2, n.state:LOOKING, n.leader:2,
n.round:0x4b, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2,
n.config version:0x0
Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,544 [myid:2] - INFO
[WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@376] -
Notification: my state:LEADING; n.sid:2, n.state:LEADING, n.leader:2,
n.round:0x4b, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2,
n.config version:0x0
Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,544 [myid:2] - INFO
[WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@376] -
Notification: my state:LEADING; n.sid:3, n.state:LEADING, n.leader:3,
n.round:0x4a, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2,
n.config version:0x0
Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,568 [myid:2] - INFO
[QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):ZooKeeperServer@519] -
Snapshot taken in 26 ms
node 3:
Mar 31 20:05:21 java[20867]: 2020-03-31 20:05:21,133 [myid:3] - INFO
[WorkerReceiver[myid=3]:FastLeaderElection$Messenger$WorkerReceiver@376] -
Notification: my state:LEADING; n.sid:2, n.state:LOOKING, n.leader:2,
n.round:0x4b, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2,
n.config version:0x0
Mar 31 20:05:21 java[20867]: 2020-03-31 20:05:21,134 [myid:3] - INFO
[WorkerReceiver[myid=3]:FastLeaderElection$Messenger$WorkerReceiver@376] -
Notification: my state:LEADING; n.sid:1, n.state:LOOKING, n.leader:2,
n.round:0x4b, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2,
n.config version:0x0
Mar 31 20:05:21 java[20867]: 2020-03-31 20:05:21,335 [myid:3] - INFO
[WorkerReceiver[myid=3]:FastLeaderElection$Messenger$WorkerReceiver@376] -
Notification: my state:LEADING; n.sid:2, n.state:LOOKING, n.leader:2,
n.round:0x4b, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2,
n.config version:0x0
Mar 31 20:05:21 java[20867]: 2020-03-31 20:05:21,736 [myid:3] - INFO
[WorkerReceiver[myid=3]:FastLeaderElection$Messenger$WorkerReceiver@376] -
Notification: my state:LEADING; n.sid:2, n.state:LOOKING, n.leader:2,
n.round:0x4b, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2,
n.config version:0x0
Mar 31 20:05:22 java[20867]: 2020-03-31 20:05:22,540 [myid:3] - INFO
[WorkerReceiver[myid=3]:FastLeaderElection$Messenger$WorkerReceiver@376] -
Notification: my state:LEADING; n.sid:1, n.state:LOOKING, n.leader:1,
n.round:0x4c, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2,
n.config version:0x0
Mar 31 20:05:22 java[20867]: 2020-03-31 20:05:22,544 [myid:3] - INFO
[WorkerReceiver[myid=3]:FastLeaderElection$Messenger$WorkerReceiver@376] -
Notification: my state:LEADING; n.sid:2, n.state:LOOKING, n.leader:2,
n.round:0x4b, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2,
n.config version:0x0
Mar 31 20:05:22 java[20867]: 2020-03-31 20:05:22,745 [myid:3] - INFO
[WorkerReceiver[myid=3]:FastLeaderElection$Messenger$WorkerReceiver@376] -
Notification: my state:LEADING; n.sid:1, n.state:LOOKING, n.leader:1,
n.round:0x4c, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2,
n.config version:0x0
Mar 31 20:05:23 java[20867]: 2020-03-31 20:05:23,147 [myid:3] - INFO
[WorkerReceiver[myid=3]:FastLeaderElection$Messenger$WorkerReceiver@376] -
Notification: my state:LEADING; n.sid:1, n.state:LOOKING, n.leader:1,
n.round:0x4c, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2,
n.config version:0x0
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
