It looks like there were a couple of things going on here:
1.) After a little more digging, I found that the leader of the ensemble
was listening on port 2888. Does only the leader of a ZooKeeper ensemble
listen on 2888?This seems to be the case but I didn't see it in the
documentation somewhere. I may have missed it.
I found that only one of my servers, the leader, was listening on port
2888. When I shut it down, another server in the ensemble started
listening on port 2888.
2.) While I did check that iptables was clear off all rules, it appears
that firewalld did have some rules that were blocking some of the
connectivity. When I added the appropriate rules for 2888/tcp and
3888/tcp, the ensemble is able to communicate as expected.
I ran in to a red herring while diagnosing the issue. I wanted to mention
it here for the record. I was receiving a
"java.net.NoRouteToHostException: No route to host" exception before
shutting down the firewall. This was a little misleading as I knew there
was a route between the VMs, they're on the same physical host and I could
SSH from one to the other. Tracing the route with traceroute proved it.
The ncat command yielded the same message, no route, when trying to connect
to these ports. It was this fact that pointed me to something going on
with a firewall somewhere.
At any rate, problem solved. Thanks for your help, Enrico.
Brian
Quoting Enrico Olivelli <eolive...@gmail.com>:
With a "grep 2888" you do not see any reference to 2888 ?
Can you try with JDK11 ?
I am sure that ZK works with JDK15 (I have used it in production) and
JDK16-ea
Enrico
Il giorno lun 8 feb 2021 alle ore 07:44 Brian @ Rickabaugh.Net <
br...@rickabaugh.net> ha scritto:
I am using 3 different virtual machines. One ZooKeeper server on each
VM. None of the hosts are listening on port 2888 and they each report
the
same error when trying to contact each other.
I have placed the id file on each of the servers in the data directory.
They each appear to know their individual ids.
I disabled SELinux and the local firewall on each of the hosts and the
issue remains.
Brian
On Feb 8, 2021, at 1:04 AM, Enrico Olivelli <eolive...@gmail.com> wrote:
Brian
Il Lun 8 Feb 2021, 03:50 Brian Rickabaugh <apache....@rickabaugh.net> ha
scritto:
Hello, everyone.
I am working on configuring a new set of 3 ZooKeeper servers and I am
finding that none of the servers are listening on port 2888. There are
no
relevant messages or exceptions logged during start up.
The servers are listening on port 3888. The servers do appear to start
up
properly and connect to each other on port 3888.
After a few seconds, I start seeing exceptions logged along the lines of
the following:
2021-02-07 21:30:16,583 [myid:1] - WARN
[QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):Follower@129
]
- Exception when following the leader
java.io.IOException: Failed connect to box02/10.1.1.202:2888
at
org.apache.zookeeper.server.quorum.Learner.connectToLeader(Learner.java:298)
at
org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:87)
at
org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1458)
I am using ZooKeeper 3.6.2, Fedora 33 and the following Java runtime.
Are you using 3 virtual machines/servers or are you running everything
in
the same machine?
Did you write the 'id' file in the data directory for each server?
Enrico
openjdk version "15.0.1" 2020-10-20
OpenJDK Runtime Environment 20.9 (build 15.0.1+9)
OpenJDK 64-Bit Server VM 20.9 (build 15.0.1+9, mixed mode, sharing)
My ZooKeeper conf file is as follows:
-----
tickTime=2000
dataDir=/opt/zookeeper-data/
clientPort=2181
initLimit=5
syncLimit=2
server.1=box00:2888:3888
server.2=box01:2888:3888
server.3=box02:2888:3888
-----
Any thoughts or ideas?
Thanks in advance, Brian
