Seems others have faced this same problem: https://issues.apache.org/jira/browse/ZOOKEEPER-3577
/Ryan On 11/18/22, 1:26 PM, "Ruel, Ryan" <[email protected]> wrote: Enrico, Sure, I can create a PR for any documentation change suggestions. I am still having some trouble with this, though. Using the dynamic reconfiguration command within the CLI tool, I was able to specify a client port of 2281 in the server specification (the same port as what I have in secureClientPort in zookeeper.conf). This was accepted by the quorum and was working just fine. However, after installing a new ZooKeeper node and initially configuring it, it's unable to bind to 2281. I'm running the most recent stable release (3.7.3). My configuration is as follows: zookeeper.conf (I have removed the irrelevant bits for brevity): secureClientPort=2281 serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory ssl.quorum.keyStore.location=/etc/zookeeper/keys/quorum/keyStore.jks sslQuorum=true X509AuthenticationProvider.superUser=SUPERUSER ssl.quorum.trustStore.location=/etc/zookeeper/keys/quorum/trustStore.jks authProvider.<our application>=<our custom authentication provider> reconfigEnabled=true ssl.trustStore.password=<our password> dynamicConfigFile=/etc/zookeeper/zookeeper.conf.dynamic.3b00000019 zookeeper.conf.dynamic.3b00000019: server.1=100.80.2.1:2888:3888:participant;0.0.0.0:2281 server.2=100.80.2.2:2888:3888:participant;0.0.0.0:2281 server.3=100.80.2.3:2888:3888:participant;0.0.0.0:2281 logs: 2022-11-18 18:07:26,990 [main] INFO server.NettyServerCnxnFactory - bound to port 2281 2022-11-18 18:07:26,990 [main] INFO server.NettyServerCnxnFactory - binding to port 0.0.0.0/0.0.0.0:2281 2022-11-18 18:07:26,995 [main] ERROR quorum.QuorumPeerMain - Unexpected exception, exiting abnormally java.net.BindException: Address already in use at java.base/sun.nio.ch.Net.bind0(Native Method) at java.base/sun.nio.ch.Net.bind(Net.java:459) at java.base/sun.nio.ch.Net.bind(Net.java:448) at java.base/sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:227) at io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:141) at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:562) at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:506) at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:491) at io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:973) at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:260) at io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:356) at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:503) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:829) So, it seems like it's trying to bind twice to port 2281 using Netty. If I remove "secureClientPort" from zookeeper.conf and restart, then ZooKeeper starts up, but as you can see in the following logs connections exception out (presumably because TLS is disabled): 2022-11-18 18:23:57,851 [nioEventLoopGroup-4-1] WARN server.NettyServerCnxn - Closing connection to /100.80.2.5:36609 java.io.IOException: Len error 369296129 at org.apache.zookeeper.server.NettyServerCnxn.receiveMessage(NettyServerCnxn.java:521) at org.apache.zookeeper.server.NettyServerCnxn.processMessage(NettyServerCnxn.java:374) at org.apache.zookeeper.server.NettyServerCnxnFactory$CnxnChannelHandler.channelRead(NettyServerCnxnFactory.java:357) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) /Ryan On 11/17/22, 8:16 AM, "Enrico Olivelli" <[email protected]> wrote: Ruel, Il giorno mer 16 nov 2022 alle ore 16:15 Ruel, Ryan <[email protected]> ha scritto: > > It seems that specifying the SECURE client port in the reconfig command does work, while also keeping the same port defined as "secureClientPort" in zookeeper.conf. > > (I thought I had tried this, but may have missed this combination) > > In any case, some clarification within the documentation may be helpful! Would you like to send a PR to add these clarifications? Thanks Enrico > > /Ryan > > On 11/15/22, 10:10 AM, "Ruel, Ryan" <[email protected]> wrote: > > In my ZooKeeper setup, I am strictly using TLS for both client and quorum communication. > > In zookeeper.conf, I have “secureClientPort=2281” defined, and do not have any “clientPort” option set. > > In the 3.8.0 documentation on dynamic reconfiguration (https://urldefense.com/v3/__https://zookeeper.apache.org/doc/r3.8.0/zookeeperReconfig.html__;!!GjvTz_vk!T5lRlM3A1syL82ZMBR5kWiVdxaCcKIlRhDY-6muCLvhez9gyJTYbbhMkWMiee0evVrX0MQc-eYwvZhl1$ ), the documentation says that the old “clientPort” configuration option should not be specified, and instead the new server keyword specification should look like this: > > server.<positive id> = <address1>:<port1>:<port2>[:role];[<client port address>:]<client port>** > > However, this specification doesn’t consider the secure client port from what I can tell. > > In some cases where the server keyword is used, I can just eliminate putting in the client port address and client port, such as within the quorum peer configuration (in zookeeper.conf or within the dynamic configuration file). > > In other cases, however, such as using the “reconfig” command in the ZK cli utility, the client port MUST be specified, or a “bad argument” type error is produced. > > I of course don’t want to put a dummy port number in the server specification which would then enable insecure communication. > > What’s the recommendation for using secure communication only while also using dynamic reconfiguration? > > P.S. Another interesting bit in the documentation is the example: > server.1=125.23.63.23:2780:2783:participant;2791 > server.2=125.23.63.24:2781:2784:participant;2792 > server.3=125.23.63.25:2782:2785:participant;2793 > > In what use case would you want to use entirely different ports for each server? Or is this just a demonstration that this is possible? > > /Ryan > > > > > >
