Thanks Enrico for sharing the jira. This is great! With the below config, I'm now able to run the 4LW commands successfully, also the downstream systems that was relying on the 4LW commands started displaying the metrics. Thanks for your help.
#secureClientPort=2281 clientPort=2281 client.portUnification=True - Karthick On Mon, 30 Mar 2020 at 21:59, Enrico Olivelli <eolive...@gmail.com> wrote: > You may be interested in Port unification, contributed by Facebook: > > https://issues.apache.org/jira/browse/ZOOKEEPER-3388 > https://issues.apache.org/jira/browse/ZOOKEEPER-3371 > > Enrico > > Il giorno lun 30 mar 2020 alle ore 13:33 karthick rn > <karthick.narend...@gmail.com> ha scritto: > > > > Hi Mate, > > > > Thanks for suggesting these options in detail > > > > 1) We are already using AdminServer as an alternate to the 4LW, hopefully > > we'll look at modifying the downstream systems to use REST instead of the > > 4LW commands. > > > > 2) Added "clientPort=2181" back to the configs and tested "srvr" & other > > whitelisted 4LW commands and they all work now :) > > > > 3) When I configure the same port "2281" for both secure and unsecure > > communication with "client.portUnification=true", the JVM exits with Bind > > exception stating the "Address already in use" & unable to start ZK. > > > > For short term, I think we'd run a mixed-mode communication like you > > mentioned in option 2 & whitelist only specific 4LW commands required and > > not all. > > > > Appreciate if someone can confirm if the 4LW is expected to work against > > secure client port or not so we can update the doc accordingly. Thanks > > again! > > > > Regards, > > Karthick > > > > On Mon, 30 Mar 2020 at 09:30, Szalay-Bekő Máté < > szalay.beko.m...@gmail.com> > > wrote: > > > > > Hi Karthick, > > > > > > I am not sure if "echo srvr | nc localhost 2281" is expected to work > > > against the secure client port. I don't think so, but maybe others know > > > better. I think you have the following options: > > > > > > 1) use the admin server which is a HTTP interface where the 4LW > commands > > > are available on a REST protocol (see > > > > https://zookeeper.apache.org/doc/r3.6.0/zookeeperAdmin.html#sc_adminserver > > > ) > > > > > > 2) if AdminServer is not an option for you, then you can configure > > > ZooKeeper to use both secure and unsecure ports. And use the unsecure > port > > > for 4LW commands, while use the secure port for the rest of the > traffic. > > > E.g.: > > > clientPort=2281 > > > secureClientPort=2282 > > > > > > 3) you can even configure ZooKeeper to use the same port for both TLS > and > > > unsecure communication. I haven't used 4LW commands with port > unification, > > > but I assume it works: > > > client.portUnification=true > > > > > > I hope some of these options will work for you. > > > > > > Kind regards, > > > Mate > > > > > > On Mon, Mar 30, 2020 at 12:24 AM karthick rn < > karthick.narend...@gmail.com > > > > > > > wrote: > > > > > > > Hello, > > > > > > > > After configuring TLS, running "echo srvr | nc localhost 2281" or any > > > other > > > > 4LW doesn’t show any output. The below messages are printed on the > ZK log > > > > whilst running the ‘srvr’ command. Also tried adding > > > > "4lw.commands.whitelist=*" to zoo.cfg but still no difference. > However, > > > > disabling TLS I'm able to see all 4LW working as expected. > > > > > > > > Let me know if this is a known issue when TLS is enabled? I'm using > ZK > > > v3.6 > > > > and have seen the same behaviour with v3.5.6 & 3.5.7. > > > > > > > > I have shared my Quorum TLS configs at the bottom, in-case if you > want to > > > > check if I'm missing something. Many thanks > > > > > > > > > > > > zookeeper.log: > > > > > > > > > > > > 2020-03-29 21:09:27,079 [myid:1] - ERROR > > > > [nioEventLoopGroup-4-1:NettyServerCnxnFactory$CertificateVerifier@434] > - > > > > Unsuccessful handshake with session 0x0 > > > > > > > > 2020-03-29 21:09:27,083 [myid:1] - WARN > > > > [nioEventLoopGroup-4-1:NettyServerCnxnFactory$CnxnChannelHandler@273] > - > > > > Exception caught > > > > > > > > io.netty.handler.codec.DecoderException: > > > > io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: > > > > 737276720a > > > > > > > > at > > > > > > > > > > > > io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) > > > > > > > > at > > > > > > > > > > > > io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) > > > > > > > > at > > > > > > > > > > > > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) > > > > > > > > at > > > > > > > > > > > > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) > > > > > > > > at > > > > > > > > > > > > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) > > > > > > > > at > > > > > > > > > > > > io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) > > > > > > > > at > > > > > > > > > > > > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) > > > > > > > > at > > > > > > > > > > > > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) > > > > > > > > at > > > > > > > > > > > > io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) > > > > > > > > at > > > > > > > > > > > > io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) > > > > > > > > at > > > > > > > > io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) > > > > > > > > at > > > > > > > > > > > > io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) > > > > > > > > at > > > > > > > > > > > > io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) > > > > > > > > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) > > > > > > > > at > > > > > > > > > > > > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) > > > > > > > > at > > > > > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > > > > > > > > at > > > > > > > > > > > > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > > > > > > > > at java.base/java.lang.Thread.run(Thread.java:834) > > > > > > > > Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS > > > > record: 737276720a > > > > > > > > at > > > > > io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1198) > > > > > > > > at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1266) > > > > > > > > at > > > > > > > > > > > > io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498) > > > > > > > > at > > > > > > > > > > > > io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437) > > > > > > > > ... 17 more > > > > > > > > > > > > > > > > conf/zoo.cfg: > > > > > > > > > > > > > > > > # Server configuration > > > > > > > > secureClientPort=2281 > > > > > > > > serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory > > > > > > > > > > > > > > > > # Quorum configuration > > > > > > > > sslQuorum=true > > > > > > > > ssl.quorum.keyStore.location=</path/to/keystore.jks> > > > > > > > > ssl.quorum.keyStore.password=<password> > > > > > > > > ssl.quorum.trustStore.location=</path/to/truststore.jks> > > > > > > > > ssl.quorum.trustStore.password=<password> > > > > > > > > > > > > > > > > bin/zkEnv.sh > > > > > > > > > > > > > > > > > > > > > > > > SERVER_JVMFLAGS="-Dzookeeper.serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory > > > > \ > > > > > > > > -Dzookeeper.ssl.keyStore.location=</path/to/keystore.jks> \ > > > > > > > > -Dzookeeper.ssl.keyStore.password=<password>\ > > > > > > > > -Dzookeeper.ssl.trustStore.location=</path/to/truststore.jks> \ > > > > > > > > -Dzookeeper.ssl.trustStore.password=<password>" > > > > > > > > > > > > > > > > > > > > > > > > CLIENT_JVMFLAGS="-Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty > > > > \ > > > > > > > > -Dzookeeper.client.secure=true \ > > > > > > > > -Dzookeeper.ssl.keyStore.location=</path/to/keystore.jks> \ > > > > > > > > -Dzookeeper.ssl.keyStore.password=<password>\ > > > > > > > > -Dzookeeper.ssl.trustStore.location=</path/to/truststore.jks> \ > > > > > > > > -Dzookeeper.ssl.trustStore.password=<password>" > > > > > > > > > > > > > > > > - Karthick > > > > > > > >
