I tried manually stopping/starting the zk with zkServer.sh stop/start and
then the zookeeper.out started showing logs again.
*# cat /logs/zookeeper.out *
2020-05-04 16:05:53,186 [myid:] - INFO [main:QuorumPeerConfig@173] -
Reading configuration from: /conf/zoo.cfg
2020-05-04 16:05:53,199 [myid:] - INFO [main:QuorumPeerConfig@459] -
clientPortAddress is 0.0.0.0:2181
2020-05-04 16:05:53,200 [myid:] - INFO [main:QuorumPeerConfig@472] -
secureClientPortAddress is 0.0.0.0:2281
2020-05-04 16:05:53,203 [myid:] - INFO [main:X509Util@77] - Setting -D
jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated
TLS renegotiation
2020-05-04 16:05:53,204 [myid:] - INFO [main:QuorumPeerConfig@479] -
observerMasterPort is not set
2020-05-04 16:05:53,204 [myid:] - INFO [main:QuorumPeerConfig@496] -
metricsProvider.className is
org.apache.zookeeper.metrics.impl.DefaultMetricsProvider
2020-05-04 16:05:53,264 [myid:2] - INFO [main:DatadirCleanupManager@78] -
autopurge.snapRetainCount set to 10
2020-05-04 16:05:53,265 [myid:2] - INFO [main:DatadirCleanupManager@79] -
autopurge.purgeInterval set to 24
2020-05-04 16:05:53,266 [myid:2] - INFO
[PurgeTask:DatadirCleanupManager$PurgeTask@139] - Purge task started.
2020-05-04 16:05:53,271 [myid:2] - INFO [main:ManagedUtil@44] - Log4j 1.2
jmx support found and enabled.
2020-05-04 16:05:53,274 [myid:2] - INFO [PurgeTask:FileTxnSnapLog@124] -
zookeeper.snapshot.trust.empty : true
2020-05-04 16:05:53,282 [myid:2] - INFO [main:QuorumPeerMain@151] -
Starting quorum peer
2020-05-04 16:05:53,292 [myid:2] - INFO [PurgeTask:SnapStream@61] -
zookeeper.snapshot.compression.method = CHECKED
2020-05-04 16:05:53,309 [myid:2] - INFO [main:ServerMetrics@62] -
ServerMetrics initialized with provider
org.apache.zookeeper.metrics.impl.DefaultMetricsProvider@5594a1b5
2020-05-04 16:05:53,321 [myid:2] - INFO
[PurgeTask:DatadirCleanupManager$PurgeTask@145] - Purge task completed.
2020-05-04 16:05:53,425 [myid:2] - INFO [main:NettyServerCnxnFactory@483] -
zookeeper.client.portUnification=true
2020-05-04 16:05:53,427 [myid:2] - INFO [main:NettyServerCnxnFactory@495] -
zookeeper.netty.advancedFlowControl.enabled = false
2020-05-04 16:05:53,427 [myid:2] - INFO [main:NettyServerCnxnFactory@105] -
handshakeThrottlingEnabled = false,
zookeeper.netty.server.outstandingHandshake.limit = -1
2020-05-04 16:05:53,485 [myid:2] - INFO [main:ServerCnxnFactory@169] -
Using org.apache.zookeeper.server.NettyServerCnxnFactory as server
connection factory
2020-05-04 16:05:53,487 [myid:2] - WARN [main:ServerCnxnFactory@309] -
maxCnxns is not configured, using default value 0.
2020-05-04 16:05:53,487 [myid:2] - INFO [main:NettyServerCnxnFactory@483] -
zookeeper.client.portUnification=true
2020-05-04 16:05:53,488 [myid:2] - INFO [main:NettyServerCnxnFactory@495] -
zookeeper.netty.advancedFlowControl.enabled = false
2020-05-04 16:05:53,488 [myid:2] - INFO [main:NettyServerCnxnFactory@105] -
handshakeThrottlingEnabled = false,
zookeeper.netty.server.outstandingHandshake.limit = -1
2020-05-04 16:05:53,489 [myid:2] - INFO [main:ServerCnxnFactory@169] -
Using org.apache.zookeeper.server.NettyServerCnxnFactory as server
connection factory
2020-05-04 16:05:53,489 [myid:2] - WARN [main:ServerCnxnFactory@309] -
maxCnxns is not configured, using default value 0.
2020-05-04 16:05:53,495 [myid:2] - INFO [main:QuorumPeer@752] -
zookeeper.quorumCnxnTimeoutMs=-1
2020-05-04 16:05:53,498 [myid:2] - INFO [main:FileTxnSnapLog@124] -
zookeeper.snapshot.trust.empty : true
2020-05-04 16:05:53,498 [myid:2] - INFO [main:QuorumPeer@1680] - Local
sessions disabled
2020-05-04 16:05:53,499 [myid:2] - INFO [main:QuorumPeer@1691] - Local
session upgrading disabled
2020-05-04 16:05:53,499 [myid:2] - INFO [main:QuorumPeer@1658] - tickTime
set to 2000
2020-05-04 16:05:53,499 [myid:2] - INFO [main:QuorumPeer@1702] -
minSessionTimeout set to 4000
2020-05-04 16:05:53,499 [myid:2] - INFO [main:QuorumPeer@1713] -
maxSessionTimeout set to 40000
2020-05-04 16:05:53,499 [myid:2] - INFO [main:QuorumPeer@1738] - initLimit
set to 10
2020-05-04 16:05:53,499 [myid:2] - INFO [main:QuorumPeer@1920] - syncLimit
set to 5
2020-05-04 16:05:53,499 [myid:2] - INFO [main:QuorumPeer@1935] -
connectToLearnerMasterLimit set to 0
2020-05-04 16:05:53,513 [myid:2] - INFO [main:ZookeeperBanner@42] -
2020-05-04 16:05:53,513 [myid:2] - INFO [main:ZookeeperBanner@42] -
______ _
2020-05-04 16:05:53,513 [myid:2] - INFO [main:ZookeeperBanner@42] - |___
/ | |
2020-05-04 16:05:53,513 [myid:2] - INFO [main:ZookeeperBanner@42] - / /
___ ___ | | __ ___ ___ _ __ ___ _ __
2020-05-04 16:05:53,513 [myid:2] - INFO [main:ZookeeperBanner@42] - / /
/ _ \ / _ \ | |/ / / _ \ / _ \ | '_ \ / _ \ | '__|
2020-05-04 16:05:53,513 [myid:2] - INFO [main:ZookeeperBanner@42] - / /__
| (_) | | (_) | | < | __/ | __/ | |_) | | __/ | |
2020-05-04 16:05:53,513 [myid:2] - INFO [main:ZookeeperBanner@42] -
/_____| \___/ \___/ |_|\_\ \___| \___| | .__/ \___| |_|
2020-05-04 16:05:53,513 [myid:2] - INFO [main:ZookeeperBanner@42] -
| |
2020-05-04 16:05:53,514 [myid:2] - INFO [main:ZookeeperBanner@42] -
|_|
2020-05-04 16:05:53,514 [myid:2] - INFO [main:ZookeeperBanner@42] -
2020-05-04 16:05:53,515 [myid:2] - INFO [main:Environment@98] - Server
environment:zookeeper.version=3.6.1--104dcb3e3fb464b30c5186d229e00af9f332524b,
built on 04/21/2020 15:01 GMT
2020-05-04 16:05:53,516 [myid:2] - INFO [main:Environment@98] - Server
environment:host.name=zoo2
2020-05-04 16:05:53,516 [myid:2] - INFO [main:Environment@98] - Server
environment:java.version=1.8.0_252
2020-05-04 16:05:53,516 [myid:2] - INFO [main:Environment@98] - Server
environment:java.vendor=Oracle Corporation
2020-05-04 16:05:53,516 [myid:2] - INFO [main:Environment@98] - Server
environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.252.b09-2.el7_8.x86_64/jre
2020-05-04 16:05:53,516 [myid:2] - INFO [main:Environment@98] - Server
environment:java.class.path=/apache-zookeeper-3.6.1-bin/bin/../zookeeper-server/target/classes:/apache-zookeeper-3.6.1-bin/bin/../build/classes:/apache-zookeeper-3.6.1-bin/bin/../zookeeper-server/target/lib/*.jar:/apache-zookeeper-3.6.1-bin/bin/../build/lib/*.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/zookeeper-prometheus-metrics-3.6.1.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/zookeeper-jute-3.6.1.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/zookeeper-3.6.1.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/snappy-java-1.1.7.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/slf4j-log4j12-1.7.25.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/slf4j-api-1.7.25.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/simpleclient_servlet-0.6.0.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/simpleclient_hotspot-0.6.0.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/simpleclient_common-0.6.0.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/simpleclient-0.6.0.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-transport-native-unix-common-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-transport-native-epoll-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-transport-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-resolver-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-handler-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-common-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-codec-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-buffer-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/metrics-core-3.2.5.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/log4j-1.2.17.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/json-simple-1.1.1.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jline-2.11.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jetty-util-9.4.24.v20191120.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jetty-servlet-9.4.24.v20191120.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jetty-server-9.4.24.v20191120.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jetty-security-9.4.24.v20191120.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jetty-io-9.4.24.v20191120.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jetty-http-9.4.24.v20191120.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/javax.servlet-api-3.1.0.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jackson-databind-2.10.3.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jackson-core-2.10.3.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jackson-annotations-2.10.3.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/commons-lang-2.6.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/commons-cli-1.2.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/audience-annotations-0.5.0.jar:/apache-zookeeper-3.6.1-bin/bin/../zookeeper-3.6.1.jar:/apache-zookeeper-3.6.1-bin/bin/../zookeeper-server/src/main/resources/lib/*.jar:/conf:
2020-05-04 16:05:53,516 [myid:2] - INFO [main:Environment@98] - Server
environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2020-05-04 16:05:53,516 [myid:2] - INFO [main:Environment@98] - Server
environment:java.io.tmpdir=/tmp
2020-05-04 16:05:53,516 [myid:2] - INFO [main:Environment@98] - Server
environment:java.compiler=<NA>
2020-05-04 16:05:53,516 [myid:2] - INFO [main:Environment@98] - Server
environment:os.name=Linux
2020-05-04 16:05:53,517 [myid:2] - INFO [main:Environment@98] - Server
environment:os.arch=amd64
2020-05-04 16:05:53,517 [myid:2] - INFO [main:Environment@98] - Server
environment:os.version=3.10.0-327.28.3.el7.x86_64
2020-05-04 16:05:53,517 [myid:2] - INFO [main:Environment@98] - Server
environment:user.name=root
2020-05-04 16:05:53,517 [myid:2] - INFO [main:Environment@98] - Server
environment:user.home=/root
2020-05-04 16:05:53,517 [myid:2] - INFO [main:Environment@98] - Server
environment:user.dir=/apache-zookeeper-3.6.1-bin
2020-05-04 16:05:53,517 [myid:2] - INFO [main:Environment@98] - Server
environment:os.memory.free=47MB
2020-05-04 16:05:53,517 [myid:2] - INFO [main:Environment@98] - Server
environment:os.memory.max=889MB
2020-05-04 16:05:53,517 [myid:2] - INFO [main:Environment@98] - Server
environment:os.memory.total=57MB
2020-05-04 16:05:53,517 [myid:2] - INFO [main:ZooKeeperServer@128] -
zookeeper.enableEagerACLCheck = false
2020-05-04 16:05:53,517 [myid:2] - INFO [main:ZooKeeperServer@136] -
zookeeper.digest.enabled = true
2020-05-04 16:05:53,518 [myid:2] - INFO [main:ZooKeeperServer@140] -
zookeeper.closeSessionTxn.enabled = true
2020-05-04 16:05:53,518 [myid:2] - INFO [main:ZooKeeperServer@1434] -
zookeeper.flushDelay=0
2020-05-04 16:05:53,518 [myid:2] - INFO [main:ZooKeeperServer@1443] -
zookeeper.maxWriteQueuePollTime=0
2020-05-04 16:05:53,518 [myid:2] - INFO [main:ZooKeeperServer@1452] -
zookeeper.maxBatchSize=1000
2020-05-04 16:05:53,518 [myid:2] - INFO [main:ZooKeeperServer@241] -
zookeeper.intBufferStartingSizeBytes = 1024
2020-05-04 16:05:53,523 [myid:2] - INFO [main:WatchManagerFactory@42] -
Using org.apache.zookeeper.server.watch.WatchManager as watch manager
2020-05-04 16:05:53,523 [myid:2] - INFO [main:WatchManagerFactory@42] -
Using org.apache.zookeeper.server.watch.WatchManager as watch manager
2020-05-04 16:05:53,524 [myid:2] - INFO [main:ZKDatabase@132] -
zookeeper.snapshotSizeFactor = 0.33
2020-05-04 16:05:53,524 [myid:2] - INFO [main:ZKDatabase@152] -
zookeeper.commitLogCount=500
2020-05-04 16:05:53,559 [myid:2] - INFO [main:QuorumPeer@1999] - Using TLS
encrypted quorum communication
2020-05-04 16:05:53,559 [myid:2] - INFO [main:QuorumPeer@2007] - Port
unification enabled
2020-05-04 16:05:53,559 [myid:2] - INFO [main:QuorumPeer@174] -
multiAddress.enabled set to false
2020-05-04 16:05:53,559 [myid:2] - INFO [main:QuorumPeer@199] -
multiAddress.reachabilityCheckEnabled set to true
2020-05-04 16:05:53,560 [myid:2] - INFO [main:QuorumPeer@186] -
multiAddress.reachabilityCheckTimeoutMs set to 1000
2020-05-04 16:05:53,560 [myid:2] - INFO [main:QuorumPeer@2461] - QuorumPeer
communication is not secured! (SASL auth disabled)
2020-05-04 16:05:53,560 [myid:2] - INFO [main:QuorumPeer@2486] -
quorum.cnxn.threads.size set to 20
2020-05-04 16:05:53,562 [myid:2] - INFO [main:FileSnap@85] - Reading
snapshot /data/version-2/snapshot.75000000a2
2020-05-04 16:05:53,582 [myid:2] - INFO [main:DataTree@1737] - The digest
value is empty in snapshot
2020-05-04 16:05:53,604 [myid:2] - INFO [main:FileTxnSnapLog@363] - 10 txns
loaded in 6 ms
2020-05-04 16:05:53,604 [myid:2] - INFO [main:ZKDatabase@289] - Snapshot
loaded in 45 ms, highest zxid is 0x2df700000002, digest is 352305768550
2020-05-04 16:05:53,605 [myid:2] - INFO [main:NettyServerCnxnFactory@670] -
binding to port 0.0.0.0/0.0.0.0:2181
2020-05-04 16:05:53,688 [myid:2] - ERROR [main:QuorumPeerMain@113] -
Unexpected exception, exiting abnormally
java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:220)
at
io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:134)
at
io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:550)
at
io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1334)
at
io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:506)
at
io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:491)
at
io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:973)
at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:248)
at
io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:356)
at
io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
at
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
at
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
2020-05-04 16:05:53,692 [myid:2] - INFO [main:ZKAuditProvider@40] -
ZooKeeper audit is enabled.
2020-05-04 16:05:53,696 [myid:2] - ERROR [main:ServiceUtils@42] - Exiting
JVM with code 1
*# nc -vz 0.0.0.0 2181*
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to 0.0.0.0:2181.
Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds.
*# nc -vz 0.0.0.0 2281*
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to 0.0.0.0:2281.
Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds.
--
Sent from: http://zookeeper-user.578899.n2.nabble.com/
