I think the "stat" command should list all ZooKeeper client connections that are handled by the given ZooKeeper server.
I'm not sure if the ZooKeeper server-server connections are listed there. (it is based on the result of the ServerCnxFactory.getAllConnectionInfo() method, which should return client connection info) But I didn't dig deep, maybe missed something. On Wed, Feb 23, 2022 at 3:23 PM Jason Grammenos <jason.gramme...@agilitypr.com.invalid> wrote: > Hello, > > I did some more digging after Enrico's last reply (netstat) and it appears > I may be miss interpreting what the "stat" command output is telling me. > Does the stat command list all connections or only connections from > followers/leaders? > > > Jason Grammenos | Operations and Infrastructure Analyst > Pronouns: he/him > P: 613.232.7797 x1131 > Toll-free: 866.545.3745 x1131 > jason.gramme...@agilitypr.com > agilitypr.com > Learn new PR tips from our free resources. > > -----Original Message----- > From: Enrico Olivelli <eolive...@gmail.com> > Sent: February 23, 2022 8:30 AM > To: UserZooKeeper <user@zookeeper.apache.org> > Subject: Re: Help With zookeeper follower connections > > Are you able to verify, using netstat, if those connections are from the > ZooKeeper process? > You can compare the TCP port > > Enrico > > Il Mer 23 Feb 2022, 14:15 Jason Grammenos > <jason.gramme...@agilitypr.com.invalid> ha scritto: > > > I do have tools that could be opening connections. I have akhq > > (akhq.io) running on two of the hosts [host01, host04 (two nodes for > > redundancy reasons)], connecting to the kafka cluster. I also have > > telegraph running the jolokia2 agent plugin to pull monitoring/stats > > data from the hosts jmx (for both kafka and zookeeper) > > > > > > Jason Grammenos | Operations and Infrastructure Analyst > > Pronouns: he/him > > P: 613.232.7797 x1131 > > Toll-free: 866.545.3745 x1131 > > jason.gramme...@agilitypr.com > > agilitypr.com > > Learn new PR tips from our free resources. > > > > -----Original Message----- > > From: Enrico Olivelli <eolive...@gmail.com> > > Sent: February 23, 2022 8:08 AM > > To: jason.gramme...@agilitypr.com.invalid > > Cc: UserZooKeeper <user@zookeeper.apache.org> > > Subject: Re: Help With zookeeper follower connections > > > > Jason, > > Do you have other tools on those machines that could open client > > connections? > > > > Enrico > > > > Il Mer 23 Feb 2022, 13:30 Jason Grammenos > > <jason.gramme...@agilitypr.com.invalid> ha scritto: > > > > > Hello, > > > > > > > > > > > > I have a 5 node zookeeper (+ kafka) cluster. I am trying to find out > > > if the inter node connection behaviour is normal. > > > > > > I should have 1 zookeeper leader and 4 followers, and I expect that > > > all 4 followers would open connections exclusively to the leader. > > > > > > So that “echo stat | nc localhost 2181” on the leader should show me > > > 6 > > > connections: 4 from the followers and 2 from itself (one via proper > > > ip, one from localhost) > > > > > > > > > > > > Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, > > > built on 05/04/2020 15:53 GMT > > > > > > Clients: > > > > > > /host02:48610[1](queued=0,recved=1455831,sent=1455833) > > > > > > /host03:36244[1](queued=0,recved=1459769,sent=1459793) > > > > > > /host05:53680[1](queued=0,recved=484,sent=484) > > > > > > /host01:48978[1](queued=0,recved=226,sent=226) > > > > > > /host04:44810[1](queued=0,recved=52,sent=52) > > > > > > /127.0.0.1:41434[0](queued=0,recved=1,sent=0) > > > > > > > > > > > > But instead I have ended up with a few connections on each node, > > > including the followers (as seen in below output). Some followers > > > are even connecting to themselves? The cluster looks stable, with > > > only 1 leader and 4 followers, but I just do not understand this > > > connection behaviour and would like to understand if this is normal > > > behaviour, mis configuration, a bug or something else. > > > > > > > > > > > > user@host01:~$ <jason.grammenos@ca02vleventstream01:~$> echo stat | > > > nc localhost 2181 > > > > > > Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, > > > built on 05/04/2020 15:53 GMT > > > > > > Clients: > > > > > > /host01:50768[1](queued=0,recved=6156,sent=6156) > > > > > > /127.0.0.1:35508[0](queued=0,recved=1,sent=0) > > > > > > > > > > > > Latency min/avg/max: 0/0/3 > > > > > > Received: 9855 > > > > > > Sent: 9854 > > > > > > Connections: 2 > > > > > > Outstanding: 0 > > > > > > Zxid: 0xb00000068 > > > > > > Mode: follower > > > > > > Node count: 226 > > > > > > > > > > > > user@host02:~$ <jason.grammenos@ca02vleventstream02:~$> echo stat | > > > nc localhost 2181 > > > > > > Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, > > > built on 05/04/2020 15:53 GMT > > > > > > Clients: > > > > > > /127.0.0.1:49624[0](queued=0,recved=1,sent=0) > > > > > > > > > > > > Latency min/avg/max: 0/0/0 > > > > > > Received: 3699 > > > > > > Sent: 3698 > > > > > > Connections: 1 > > > > > > Outstanding: 0 > > > > > > Zxid: 0xb00000068 > > > > > > Mode: follower > > > > > > Node count: 226 > > > > > > > > > > > > usaer@host03:~$ <jason.grammenos@ca02vleventstream03:~$> echo stat | > > > nc localhost 2181 > > > > > > Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, > > > built on 05/04/2020 15:53 GMT > > > > > > Clients: > > > > > > /host03:50674[1](queued=0,recved=6136,sent=6136) > > > > > > /127.0.0.1:54554[0](queued=0,recved=1,sent=0) > > > > > > > > > > > > Latency min/avg/max: 0/0/22 > > > > > > Received: 9795 > > > > > > Sent: 9794 > > > > > > Connections: 2 > > > > > > Outstanding: 0 > > > > > > Zxid: 0xb00000068 > > > > > > Mode: follower > > > > > > Node count: 226 > > > > > > > > > > > > user@host04:~$ <jason.grammenos@ca04vleventstream04:~$> echo stat | > > > nc localhost 2181 > > > > > > Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, > > > built on 05/04/2020 15:53 GMT > > > > > > Clients: > > > > > > /host02:42874[1](queued=0,recved=143,sent=143) > > > > > > /host04:38222[1](queued=0,recved=89,sent=89) > > > > > > /127.0.0.1:58796[0](queued=0,recved=1,sent=0) > > > > > > Latency min/avg/max: 0/0/27 > > > > > > Received: 287 > > > > > > Sent: 286 > > > > > > Connections: 3 > > > > > > Outstanding: 0 > > > > > > Zxid: 0xb00000056 > > > > > > Mode: follower > > > > > > Node count: 226 > > > > > > > > > > > > user@host05:~$ <jason.grammenos@ca04vleventstream05:~$> echo stat | > > > nc localhost 2181 > > > > > > Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, > > > built on 05/04/2020 15:53 GMT > > > > > > Clients: > > > > > > /127.0.0.1:38680[0](queued=0,recved=1,sent=0) > > > > > > /host01:41198[1](queued=0,recved=6339,sent=6340) > > > > > > > > > > > > Latency min/avg/max: 0/0/8 > > > > > > Received: 10041 > > > > > > Sent: 10041 > > > > > > Connections: 2 > > > > > > Outstanding: 0 > > > > > > Zxid: 0xb00000068 > > > > > > Mode: leader > > > > > > Node count: 226 > > > > > > Proposal sizes last/min/max: 220/36/392 > > > > > > > > > > > > > > > > > > Thank you > > > > > > Jason > > > > > > > > > > > > Jason Grammenos | Operations and Infrastructure Analyst > > > > > > Pronouns: he/him > > > P: 613.232.7797 x1131 > > > Toll-free: 866.545.3745 x1131 > > > jason.gramme...@agilitypr.com > > > agilitypr.com > > > > > > Learn new PR tips from our free resources > > > <https://www.agilitypr.com/resources/>. > > > > > > > > > > > >
