Hi, during testing upgrade to 3.6.1 version and using secure quorum ssl
communication my zookeeper is not starting up.
*config file:*
dataDir=/data
dataLogDir=/datalog
tickTime=2000
initLimit=10
syncLimit=5
maxClientCnxns=0
autopurge.snapRetainCount=10
autopurge.purgeInterval=24
admin.enableServer=false
snapshot.trust.empty=true
reconfigEnabled=true
audit.enable=true
clientPort=2181
secureClientPort=2281
sslQuorum=true
portUnification=true
serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory
client.portUnification=true
clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
ssl.client.enable=true
clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
ssl.quorum.keyStore.location=/apache-zookeeper-3.6.1-bin/java/node1.ks
ssl.quorum.keyStore.password=<pw>
ssl.quorum.trustStore.location=/apache-zookeeper-3.6.1-bin/java/truststore.ks
ssl.quorum.trustStore.password=<pw>
4lw.commands.whitelist=*
quorumListenOnAllIPs=true
server.2=<fqdn of zoo2>:2888:3888:participant
server.3=<fqdn of zoo2>:2888:3888:participant
server.1=<fqdn of zoo1>:2888:3888:participant
*# bin/zkServer.sh status*
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Error contacting service. It is probably not running.
*# cat /logs/zookeeper_audit.log *
2020-05-01 22:06:12,213 INFO audit.Log4jAuditLogger: user=zookeeper
operation=serverStart result=success
2020-05-01 23:08:30,859 INFO audit.Log4jAuditLogger: user=zookeeper
operation=serverStart result=success
*# bin/zkCli.sh -server 127.0.0.1:2181*
/usr/bin/java
Connecting to 127.0.0.1:2181
2020-05-01 23:19:20,035 [myid:] - INFO [main:Environment@98] - Client
environment:zookeeper.version=3.6.1--104dcb3e3fb464b30c5186d229e00af9f332524b,
built on 04/21/2020 15:01 GMT
2020-05-01 23:19:20,039 [myid:] - INFO [main:Environment@98] - Client
environment:host.name=zoo1
2020-05-01 23:19:20,039 [myid:] - INFO [main:Environment@98] - Client
environment:java.version=1.8.0_252
2020-05-01 23:19:20,042 [myid:] - INFO [main:Environment@98] - Client
environment:java.vendor=Oracle Corporation
2020-05-01 23:19:20,042 [myid:] - INFO [main:Environment@98] - Client
environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.252.b09-2.el7_8.x86_64/jre
2020-05-01 23:19:20,042 [myid:] - INFO [main:Environment@98] - Client
environment:java.class.path=/apache-zookeeper-3.6.1-bin/bin/../zookeeper-server/target/classes:/apache-zookeeper-3.6.1-bin/bin/../build/classes:/apache-zookeeper-3.6.1-bin/bin/../zookeeper-server/target/lib/*.jar:/apache-zookeeper-3.6.1-bin/bin/../build/lib/*.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/zookeeper-prometheus-metrics-3.6.1.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/zookeeper-jute-3.6.1.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/zookeeper-3.6.1.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/snappy-java-1.1.7.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/slf4j-log4j12-1.7.25.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/slf4j-api-1.7.25.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/simpleclient_servlet-0.6.0.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/simpleclient_hotspot-0.6.0.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/simpleclient_common-0.6.0.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/simpleclient-0.6.0.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-transport-native-unix-common-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-transport-native-epoll-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-transport-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-resolver-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-handler-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-common-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-codec-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-buffer-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/metrics-core-3.2.5.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/log4j-1.2.17.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/json-simple-1.1.1.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jline-2.11.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jetty-util-9.4.24.v20191120.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jetty-servlet-9.4.24.v20191120.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jetty-server-9.4.24.v20191120.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jetty-security-9.4.24.v20191120.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jetty-io-9.4.24.v20191120.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jetty-http-9.4.24.v20191120.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/javax.servlet-api-3.1.0.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jackson-databind-2.10.3.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jackson-core-2.10.3.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jackson-annotations-2.10.3.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/commons-lang-2.6.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/commons-cli-1.2.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/audience-annotations-0.5.0.jar:/apache-zookeeper-3.6.1-bin/bin/../zookeeper-3.6.1.jar:/apache-zookeeper-3.6.1-bin/bin/../zookeeper-server/src/main/resources/lib/*.jar:/conf:
2020-05-01 23:19:20,043 [myid:] - INFO [main:Environment@98] - Client
environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2020-05-01 23:19:20,043 [myid:] - INFO [main:Environment@98] - Client
environment:java.io.tmpdir=/tmp
2020-05-01 23:19:20,043 [myid:] - INFO [main:Environment@98] - Client
environment:java.compiler=<NA>
2020-05-01 23:19:20,043 [myid:] - INFO [main:Environment@98] - Client
environment:os.name=Linux
2020-05-01 23:19:20,043 [myid:] - INFO [main:Environment@98] - Client
environment:os.arch=amd64
2020-05-01 23:19:20,043 [myid:] - INFO [main:Environment@98] - Client
environment:os.version=3.10.0-327.28.3.el7.x86_64
2020-05-01 23:19:20,043 [myid:] - INFO [main:Environment@98] - Client
environment:user.name=root
2020-05-01 23:19:20,043 [myid:] - INFO [main:Environment@98] - Client
environment:user.home=/root
2020-05-01 23:19:20,044 [myid:] - INFO [main:Environment@98] - Client
environment:user.dir=/apache-zookeeper-3.6.1-bin
2020-05-01 23:19:20,044 [myid:] - INFO [main:Environment@98] - Client
environment:os.memory.free=51MB
2020-05-01 23:19:20,046 [myid:] - INFO [main:Environment@98] - Client
environment:os.memory.max=228MB
2020-05-01 23:19:20,046 [myid:] - INFO [main:Environment@98] - Client
environment:os.memory.total=57MB
2020-05-01 23:19:20,052 [myid:] - INFO [main:ZooKeeper@1005] - Initiating
client connection, connectString=127.0.0.1:2181 sessionTimeout=30000
watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@5e265ba4
2020-05-01 23:19:20,059 [myid:] - INFO [main:X509Util@77] - Setting -D
jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated
TLS renegotiation
2020-05-01 23:19:20,071 [myid:] - INFO [main:ClientCnxnSocket@239] -
jute.maxbuffer value is 1048575 Bytes
2020-05-01 23:19:20,083 [myid:] - INFO [main:ClientCnxn@1703] -
zookeeper.request.timeout value is 0. feature enabled=false
2020-05-01 23:19:20,091 [myid:127.0.0.1:2181] - INFO
[main-SendThread(127.0.0.1:2181):ClientCnxn$SendThread@1154] - Opening
socket connection to server localhost/127.0.0.1:2181.
2020-05-01 23:19:20,091 [myid:127.0.0.1:2181] - INFO
[main-SendThread(127.0.0.1:2181):ClientCnxn$SendThread@1156] - SASL config
status: Will not attempt to authenticate using SASL (unknown error)
Welcome to ZooKeeper!
JLine support is enabled
2020-05-01 23:19:20,235 [myid:127.0.0.1:2181] - INFO
[main-SendThread(127.0.0.1:2181):ClientCnxn$SendThread@986] - Socket
connection established, initiating session, client: /127.0.0.1:37360,
server: localhost/127.0.0.1:2181
2020-05-01 23:19:20,259 [myid:127.0.0.1:2181] - WARN
[main-SendThread(127.0.0.1:2181):ClientCnxn$SendThread@1272] - Session 0x0
for sever localhost/127.0.0.1:2181, Closing socket connection. Attempting
reconnect except it is a SessionExpiredException.
EndOfStreamException: Unable to read additional data from server sessionid
0x0, likely server has closed socket
at
org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:75)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:348)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1262)
[zk: 127.0.0.1:2181(CONNECTING) 0] 2020-05-01 23:19:21,591
[myid:127.0.0.1:2181] - INFO
[main-SendThread(127.0.0.1:2181):ClientCnxn$SendThread@1154] - Opening
socket connection to server localhost/127.0.0.1:2181.
2020-05-01 23:19:21,591 [myid:127.0.0.1:2181] - INFO
[main-SendThread(127.0.0.1:2181):ClientCnxn$SendThread@1156] - SASL config
status: Will not attempt to authenticate using SASL (unknown error)
2020-05-01 23:19:21,592 [myid:127.0.0.1:2181] - INFO
[main-SendThread(127.0.0.1:2181):ClientCnxn$SendThread@986] - Socket
connection established, initiating session, client: /127.0.0.1:37362,
server: localhost/127.0.0.1:2181
2020-05-01 23:19:21,598 [myid:127.0.0.1:2181] - WARN
[main-SendThread(127.0.0.1:2181):ClientCnxn$SendThread@1272] - Session 0x0
for sever localhost/127.0.0.1:2181, Closing socket connection. Attempting
reconnect except it is a SessionExpiredException.
EndOfStreamException: Unable to read additional data from server sessionid
0x0, likely server has closed socket
at
org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:75)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:348)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1262)
2020-05-01 23:19:23,303 [myid:127.0.0.1:2181] - INFO
[main-SendThread(127.0.0.1:2181):ClientCnxn$SendThread@1154] - Opening
socket connection to server localhost/127.0.0.1:2181.
2020-05-01 23:19:23,304 [myid:127.0.0.1:2181] - INFO
[main-SendThread(127.0.0.1:2181):ClientCnxn$SendThread@1156] - SASL config
status: Will not attempt to authenticate using SASL (unknown error)
2020-05-01 23:19:23,304 [myid:127.0.0.1:2181] - INFO
[main-SendThread(127.0.0.1:2181):ClientCnxn$SendThread@986] - Socket
connection established, initiating session, client: /127.0.0.1:37364,
server: localhost/127.0.0.1:2181
2020-05-01 23:19:23,313 [myid:127.0.0.1:2181] - WARN
[main-SendThread(127.0.0.1:2181):ClientCnxn$SendThread@1272] - Session 0x0
for sever localhost/127.0.0.1:2181, Closing socket connection. Attempting
reconnect except it is a SessionExpiredException.
EndOfStreamException: Unable to read additional data from server sessionid
0x0, likely server has closed socket
at
org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:75)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:348)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1262)
(it continues on like this repeating the same exception)
Don't see /logs/zookeeper.out
*# nc -vz zoo2 2181*
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to <fqdn zoo2>:2181.
Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds.
*# nc -vz zoo3 2181*
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to <fqdn zoo3>:2181.
Ncat: 0 bytes sent, 0 bytes received in 0.02 seconds.
*# nc -vz zoo2 2182*
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connection refused.
*# nc -vz zoo3 2182*
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connection refused.
*# ps -ef | grep zookeeper*
root 13 1 0 23:08 ? 00:00:00 su -m zookeeper zkServer.sh
start-foreground
zookeep+ 14 13 0 23:08 ? 00:00:03 java
-Dzookeeper.log.dir=/logs -Dzookeeper.log.file=zookeeper--server-zoo1.log
-Dzookeeper.root.logger=DEBUG -XX:+HeapDumpOnOutOfMemoryError
-XX:OnOutOfMemoryError=kill -9 %p -cp
/apache-zookeeper-3.6.1-bin/bin/../zookeeper-server/target/classes:/apache-zookeeper-3.6.1-bin/bin/../build/classes:/apache-zookeeper-3.6.1-bin/bin/../zookeeper-server/target/lib/*.jar:/apache-zookeeper-3.6.1-bin/bin/../build/lib/*.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/zookeeper-prometheus-metrics-3.6.1.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/zookeeper-jute-3.6.1.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/zookeeper-3.6.1.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/snappy-java-1.1.7.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/slf4j-log4j12-1.7.25.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/slf4j-api-1.7.25.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/simpleclient_servlet-0.6.0.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/simpleclient_hotspot-0.6.0.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/simpleclient_common-0.6.0.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/simpleclient-0.6.0.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-transport-native-unix-common-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-transport-native-epoll-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-transport-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-resolver-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-handler-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-common-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-codec-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-buffer-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/metrics-core-3.2.5.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/log4j-1.2.17.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/json-simple-1.1.1.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jline-2.11.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jetty-util-9.4.24.v20191120.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jetty-servlet-9.4.24.v20191120.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jetty-server-9.4.24.v20191120.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jetty-security-9.4.24.v20191120.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jetty-io-9.4.24.v20191120.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jetty-http-9.4.24.v20191120.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/javax.servlet-api-3.1.0.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jackson-databind-2.10.3.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jackson-core-2.10.3.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jackson-annotations-2.10.3.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/commons-lang-2.6.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/commons-cli-1.2.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/audience-annotations-0.5.0.jar:/apache-zookeeper-3.6.1-bin/bin/../zookeeper-3.6.1.jar:/apache-zookeeper-3.6.1-bin/bin/../zookeeper-server/src/main/resources/lib/*.jar:/conf:
-Xmx1000m -Xmx4g -Xms1g -Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.local.only=false
org.apache.zookeeper.server.quorum.QuorumPeerMain /conf/zoo.cfg
I think the issue is around the secureClientPort not taking the connection?
What else needs to be configured to work or would it be an issue between my
machines? At a loss for how to proceed. Any debug or configuration guidance
is appreciated.
--
Sent from: http://zookeeper-user.578899.n2.nabble.com/
