evfurman opened a new issue #10388:
URL: https://github.com/apache/pulsar/issues/10388
**Describe the bug**
Bookies unable to connect to ZK over TLS on Pulsar version `2.7.1`.
**To Reproduce**
Steps to reproduce the behavior:
1. Deploy Pulsar cluster version `2.7.1`
2. Configure certificate-based TLS/SSL cluster-wide.
3. Bookies fail to initialize metadata drive and `pulsar-admin` commands
return a 500 errror.
**Expected behavior**
Bookies should be able to connect to Zookeeper nodes over TLS/SSL without
issue.
**Additional context**
```
# BOOKIE LOGS
15:11:55.930 [main] ERROR org.apache.bookkeeper.meta.zk.ZKMetadataDriverBase
- Failed to create zookeeper client to
zk-1.pluster-271.qa.example.com:2281,zk-2.pluster-271.qa.example.com:2281,zk-3.pluster-271.qa.example.com:2281
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode
= ConnectionLoss
at
org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
at
org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase.waitForConnection(ZooKeeperWatcherBase.java:159)
at
org.apache.bookkeeper.zookeeper.ZooKeeperClient$Builder.build(ZooKeeperClient.java:260)
at
org.apache.bookkeeper.meta.zk.ZKMetadataDriverBase.initialize(ZKMetadataDriverBase.java:197)
at
org.apache.bookkeeper.meta.zk.ZKMetadataBookieDriver.initialize(ZKMetadataBookieDriver.java:60)
at
org.apache.bookkeeper.bookie.Bookie.instantiateMetadataDriver(Bookie.java:1093)
at org.apache.bookkeeper.bookie.Bookie.<init>(Bookie.java:711)
at
org.apache.bookkeeper.proto.BookieServer.newBookie(BookieServer.java:152)
at
org.apache.bookkeeper.proto.BookieServer.<init>(BookieServer.java:120)
at
org.apache.bookkeeper.server.service.BookieService.<init>(BookieService.java:52)
at org.apache.bookkeeper.server.Main.buildBookieServer(Main.java:304)
at org.apache.bookkeeper.server.Main.doMain(Main.java:226)
at org.apache.bookkeeper.server.Main.main(Main.java:208)
15:11:55.957 [main] ERROR org.apache.bookkeeper.server.Main - Failed to
build bookie server
org.apache.bookkeeper.bookie.BookieException$MetadataStoreException: Failed to
initialize metadata bookie driver
at
org.apache.bookkeeper.bookie.Bookie.instantiateMetadataDriver(Bookie.java:1103)
at org.apache.bookkeeper.bookie.Bookie.<init>(Bookie.java:711)
at
org.apache.bookkeeper.proto.BookieServer.newBookie(BookieServer.java:152)
at
org.apache.bookkeeper.proto.BookieServer.<init>(BookieServer.java:120)
at
org.apache.bookkeeper.server.service.BookieService.<init>(BookieService.java:52)
at org.apache.bookkeeper.server.Main.buildBookieServer(Main.java:304)
at org.apache.bookkeeper.server.Main.doMain(Main.java:226)
at org.apache.bookkeeper.server.Main.main(Main.java:208)
Caused by: org.apache.bookkeeper.meta.exceptions.MetadataException: Failed
to create zookeeper client to
zk-1.pluster-271.qa.example.com:2281,zk-2.pluster-271.qa.example.com:2281,zk-3.pluster-271.qa.example.com:2281
at
org.apache.bookkeeper.meta.zk.ZKMetadataDriverBase.initialize(ZKMetadataDriverBase.java:217)
at
org.apache.bookkeeper.meta.zk.ZKMetadataBookieDriver.initialize(ZKMetadataBookieDriver.java:60)
at
org.apache.bookkeeper.bookie.Bookie.instantiateMetadataDriver(Bookie.java:1093)
... 7 more
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss
at
org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
at
org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase.waitForConnection(ZooKeeperWatcherBase.java:159)
# ZOOKEEPER LOGS
18:55:53.973 [CommitProcWorkThread-2] DEBUG
org.apache.zookeeper.server.FinalRequestProcessor - Processing request::
sessionid:0x1000017dd3e0007 type:ping cxid:0xfffffffffffffffe
zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a
18:55:53.973 [CommitProcWorkThread-2] DEBUG
org.apache.zookeeper.server.FinalRequestProcessor - sessionid:0x1000017dd3e0007
type:ping cxid:0xfffffffffffffffe zxid:0xfffffffffffffffe txntype:unknown
reqpath:n/a
18:55:54.620 [epollEventLoopGroup-7-2] DEBUG
org.apache.zookeeper.server.NettyServerCnxnFactory - SSL handler added for
channel: [id: 0x66fb966c, L:/10.3.21.69:2281 - R:/10.3.22.228:35764]
18:55:54.620 [epollEventLoopGroup-7-2] ERROR
org.apache.zookeeper.server.NettyServerCnxnFactory - Unsuccessful handshake
with session 0x0
18:55:54.620 [epollEventLoopGroup-7-2] DEBUG
org.apache.zookeeper.server.NettyServerCnxn - close called for sessionid:0x0
18:55:54.620 [epollEventLoopGroup-7-2] DEBUG
org.apache.zookeeper.server.NettyServerCnxn - cnxns size:4
18:55:54.620 [epollEventLoopGroup-7-2] WARN
org.apache.zookeeper.server.NettyServerCnxnFactory - Exception caught
io.netty.handler.codec.DecoderException:
io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record:
0000002d000000000000000000000000000075300000000000000000000000100000000000000000000000000000000000
at
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:471)
at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at
io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792)
at
io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
at
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS
record:
0000002d000000000000000000000000000075300000000000000000000000100000000000000000000000000000000000
at
io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1254)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1322)
at
io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501)
at
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440)
... 15 more
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
