Hi Rounak, Your server definitions should be the same for each config file in each instance (they are not).
One of your servers is 0.0.0.0; it should be the appropriate IP address. Let me know if that helps. Josef Roehrl​ FuseForward | Senior Architect - Professional Services [https://fuseforward.atlassian.net/wiki/download/attachments/512327681/image001.png?version=1&modificationDate=1537397840684&cacheVersion=1&api=v2] Website<https://fuseforward.com/?utm_source=Email%20Signature&utm_medium=email%20signature&utm_campaign=email%20signature> | Newsletter<https://fuseforward.com/subscribe-to-our-newsletter/?utm_source=Email%20Signature&utm_medium=Email%20Signature&utm_campaign=Email%20Signature> | Twitter<https://twitter.com/fuseforward> | LinkedIn<https://www.linkedin.com/company/fuseforward/?originalSubdomain=ca> ________________________________ From: Rounak Kakkar (rkakkar) <rkak...@cisco.com.INVALID> Sent: Wednesday, July 5, 2023 11:30 PM To: u...@zookeeper.apache.org <u...@zookeeper.apache.org>; dev@zookeeper.apache.org <dev@zookeeper.apache.org> Cc: Jitesh Mulchandani (jmulchan) <jmulc...@cisco.com> Subject: Zookeeper server doesn't list all the participants Hello Team, We are using a ZK cluster with 3 nodes in the cluster (namely: vm12, vm18 and vm19). We are seeing a issue with ZK cluster, where on one of the nodes (vm18) in the cluster, ZK doesn’t list all the participants. The node on which the issue is seen is a leader and the other two nodes (vm12 and vm19) are followers and they are connected to the leader node (vm18). Could you please help us to debug this issue? ZK Version: Zookeeper version: 3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715 Node on which issue is seen (vm18). It shows only 2 participants. vm18:~# docker exec -i coordination-server /var/lib/zookeeper/bin/zkCli.sh -server localhost config | grep participant server.0=10.0.105.32:2888:3888:participant server.1=0.0.0.0:2888:3888:participant vm18:~# We can see in config there are nodes. vm18:~# echo conf | nc 10.0.105.38 2181 clientPort=2181 secureClientPort=-1 dataDir=/data/6356d4ed-ed79-4481-9913-e1c470ec2e79/version-2 dataDirSize=201387396 dataLogDir=/data/6356d4ed-ed79-4481-9913-e1c470ec2e79/version-2 dataLogSize=201387396 tickTime=2000 maxClientCnxns=0 minSessionTimeout=4000 maxSessionTimeout=40000 clientPortListenBacklog=-1 serverId=1 initLimit=5 syncLimit=2 electionAlg=3 electionPort=3888 quorumPort=2888 peerType=0 membership: server.0=10.0.105.32:2888:3888:participant server.1=0.0.0.0:2888:3888:participant server.2=10.0.105.39:2888:3888:participant version=0 And this node is a leader. vm18:~# echo stat | nc 10.0.105.38 2181 Zookeeper version: 3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715, built on 09/04/2020 12:44 GMT Clients: /10.0.105.32:33086[1](queued=0,recved=229788,sent=229790) /10.0.105.38:52350[1](queued=0,recved=55027,sent=55034) /169.254.8.3:33050[1](queued=0,recved=3337,sent=3341) /10.0.105.38:50552[0](queued=0,recved=1,sent=0) Latency min/avg/max: 0/0.2586/1626 Received: 296333 Sent: 296451 Connections: 4 Outstanding: 0 Zxid: 0x5000015b9 Mode: leader Node count: 381 Proposal Other two nodes where participants all three participants are seen. VM12: vm12:~# docker exec -i coordination-server /var/lib/zookeeper/bin/zkCli.sh -server localhost config | grep participant server.0=0.0.0.0:2888:3888:participant server.1=10.0.105.38:2888:3888:participant server.2=10.0.105.39:2888:3888:participant vm12:~# vm12:~# echo stat | nc 10.0.105.32 2181 Zookeeper version: 3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715, built on 09/04/2020 12:44 GMT Clients: /10.0.105.32:41494[0](queued=0,recved=1,sent=0) /169.254.8.3:55894[1](queued=0,recved=3037,sent=3038) /10.0.105.38:35292[1](queued=0,recved=231472,sent=231474) /10.0.105.32:39984[1](queued=0,recved=55138,sent=55148) /10.0.105.39:52614[1](queued=0,recved=5568,sent=5721) /10.0.105.32:38996[1](queued=0,recved=6219,sent=6400) /10.0.105.39:53858[0](queued=0,recved=1,sent=0) /10.0.105.39:53880[1](queued=0,recved=54945,sent=54949) Latency min/avg/max: 0/0.4517/350 Received: 361365 Sent: 361728 Connections: 8 Outstanding: 0 Zxid: 0x5000015d4 Mode: follower Node count: 384 vm12:~# VM19: vm19:~# docker exec -i coordination-server /var/lib/zookeeper/bin/zkCli.sh -server localhost config | grep participant server.0=10.0.105.32:2888:3888:participant server.1=10.0.105.38:2888:3888:participant server.2=0.0.0.0:2888:3888:participant vm19:~# vm19:~# echo stat | nc 10.0.105.39 2181 Zookeeper version: 3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715, built on 09/04/2020 12:44 GMT Clients: /169.254.8.3:58810[1](queued=0,recved=3246,sent=3250) /10.0.105.38:43338[1](queued=0,recved=4097,sent=4258) /169.254.8.1:53464[1](queued=0,recved=228908,sent=228908) /10.0.105.39:41558[0](queued=0,recved=1,sent=0) Latency min/avg/max: 0/0.3002/276 Received: 238713 Sent: 238881 Connections: 4 Outstanding: 0 Zxid: 0x5000015dc Mode: follower Node count: 386 vm19:~# Thanks, Rounak
