Re: Confusion around client ports and dynamic reconfiguration
We run secure client port at 2182 and standard port 2181. This 2181 is in the dynamic config strings. Then we have a firewalld rule to block incoming traffic on 2181. For us this works, no problem with Curator Ensemble tracker either if I recall. Our Curator based clients connect fine on the secure port and we run frequent dynamic reconfiguration. Regards Chris On 18 November 2022 20:15:39 "Ruel, Ryan" wrote: Seems others have faced this same problem: https://issues.apache.org/jira/browse/ZOOKEEPER-3577 /Ryan On 11/18/22, 1:26 PM, "Ruel, Ryan" wrote: Enrico, Sure, I can create a PR for any documentation change suggestions. I am still having some trouble with this, though. Using the dynamic reconfiguration command within the CLI tool, I was able to specify a client port of 2281 in the server specification (the same port as what I have in secureClientPort in zookeeper.conf). This was accepted by the quorum and was working just fine. However, after installing a new ZooKeeper node and initially configuring it, it's unable to bind to 2281. I'm running the most recent stable release (3.7.3). My configuration is as follows: zookeeper.conf (I have removed the irrelevant bits for brevity): secureClientPort=2281 serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory ssl.quorum.keyStore.location=/etc/zookeeper/keys/quorum/keyStore.jks sslQuorum=true X509AuthenticationProvider.superUser=SUPERUSER ssl.quorum.trustStore.location=/etc/zookeeper/keys/quorum/trustStore.jks authProvider.= reconfigEnabled=true ssl.trustStore.password= dynamicConfigFile=/etc/zookeeper/zookeeper.conf.dynamic.3b0019 zookeeper.conf.dynamic.3b0019: server.1=100.80.2.1:2888:3888:participant;0.0.0.0:2281 server.2=100.80.2.2:2888:3888:participant;0.0.0.0:2281 server.3=100.80.2.3:2888:3888:participant;0.0.0.0:2281 logs: 2022-11-18 18:07:26,990 [main] INFO server.NettyServerCnxnFactory - bound to port 2281 2022-11-18 18:07:26,990 [main] INFO server.NettyServerCnxnFactory - binding to port 0.0.0.0/0.0.0.0:2281 2022-11-18 18:07:26,995 [main] ERROR quorum.QuorumPeerMain - Unexpected exception, exiting abnormally java.net.BindException: Address already in use at java.base/sun.nio.ch.Net.bind0(Native Method) at java.base/sun.nio.ch.Net.bind(Net.java:459) at java.base/sun.nio.ch.Net.bind(Net.java:448) at java.base/sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:227) at io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:141) at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:562) at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:506) at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:491) at io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:973) at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:260) at io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:356) at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:503) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:829) So, it seems like it's trying to bind twice to port 2281 using Netty. If I remove "secureClientPort" from zookeeper.conf and restart, then ZooKeeper starts up, but as you can see in the following logs connections exception out (presumably because TLS is disabled): 2022-11-18 18:23:57,851 [nioEventLoopGroup-4-1] WARN server.NettyServerCnxn - Closing connection to /100.80.2.5:36609 java.io.IOException: Len error 369296129 at org.apache.zookeeper.server.NettyServerCnxn.receiveMessage(NettyServerCnxn.java:521) at org.apache.zookeeper.server.NettyServerCnxn.processMessage(NettyServerCnxn.java:374) at org.apache.zookeeper.server.NettyServerCnxnFactory$CnxnChannelHandler.channelRead(NettyServerCnxnFactory.java:357) at
Re: Confusion around client ports and dynamic reconfiguration
Seems others have faced this same problem: https://issues.apache.org/jira/browse/ZOOKEEPER-3577 /Ryan On 11/18/22, 1:26 PM, "Ruel, Ryan" wrote: Enrico, Sure, I can create a PR for any documentation change suggestions. I am still having some trouble with this, though. Using the dynamic reconfiguration command within the CLI tool, I was able to specify a client port of 2281 in the server specification (the same port as what I have in secureClientPort in zookeeper.conf). This was accepted by the quorum and was working just fine. However, after installing a new ZooKeeper node and initially configuring it, it's unable to bind to 2281. I'm running the most recent stable release (3.7.3). My configuration is as follows: zookeeper.conf (I have removed the irrelevant bits for brevity): secureClientPort=2281 serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory ssl.quorum.keyStore.location=/etc/zookeeper/keys/quorum/keyStore.jks sslQuorum=true X509AuthenticationProvider.superUser=SUPERUSER ssl.quorum.trustStore.location=/etc/zookeeper/keys/quorum/trustStore.jks authProvider.= reconfigEnabled=true ssl.trustStore.password= dynamicConfigFile=/etc/zookeeper/zookeeper.conf.dynamic.3b0019 zookeeper.conf.dynamic.3b0019: server.1=100.80.2.1:2888:3888:participant;0.0.0.0:2281 server.2=100.80.2.2:2888:3888:participant;0.0.0.0:2281 server.3=100.80.2.3:2888:3888:participant;0.0.0.0:2281 logs: 2022-11-18 18:07:26,990 [main] INFO server.NettyServerCnxnFactory - bound to port 2281 2022-11-18 18:07:26,990 [main] INFO server.NettyServerCnxnFactory - binding to port 0.0.0.0/0.0.0.0:2281 2022-11-18 18:07:26,995 [main] ERROR quorum.QuorumPeerMain - Unexpected exception, exiting abnormally java.net.BindException: Address already in use at java.base/sun.nio.ch.Net.bind0(Native Method) at java.base/sun.nio.ch.Net.bind(Net.java:459) at java.base/sun.nio.ch.Net.bind(Net.java:448) at java.base/sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:227) at io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:141) at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:562) at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:506) at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:491) at io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:973) at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:260) at io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:356) at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:503) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:829) So, it seems like it's trying to bind twice to port 2281 using Netty. If I remove "secureClientPort" from zookeeper.conf and restart, then ZooKeeper starts up, but as you can see in the following logs connections exception out (presumably because TLS is disabled): 2022-11-18 18:23:57,851 [nioEventLoopGroup-4-1] WARN server.NettyServerCnxn - Closing connection to /100.80.2.5:36609 java.io.IOException: Len error 369296129 at org.apache.zookeeper.server.NettyServerCnxn.receiveMessage(NettyServerCnxn.java:521) at org.apache.zookeeper.server.NettyServerCnxn.processMessage(NettyServerCnxn.java:374) at org.apache.zookeeper.server.NettyServerCnxnFactory$CnxnChannelHandler.channelRead(NettyServerCnxnFactory.java:357) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
Re: Confusion around client ports and dynamic reconfiguration
Enrico, Sure, I can create a PR for any documentation change suggestions. I am still having some trouble with this, though. Using the dynamic reconfiguration command within the CLI tool, I was able to specify a client port of 2281 in the server specification (the same port as what I have in secureClientPort in zookeeper.conf). This was accepted by the quorum and was working just fine. However, after installing a new ZooKeeper node and initially configuring it, it's unable to bind to 2281. I'm running the most recent stable release (3.7.3). My configuration is as follows: zookeeper.conf (I have removed the irrelevant bits for brevity): secureClientPort=2281 serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory ssl.quorum.keyStore.location=/etc/zookeeper/keys/quorum/keyStore.jks sslQuorum=true X509AuthenticationProvider.superUser=SUPERUSER ssl.quorum.trustStore.location=/etc/zookeeper/keys/quorum/trustStore.jks authProvider.= reconfigEnabled=true ssl.trustStore.password= dynamicConfigFile=/etc/zookeeper/zookeeper.conf.dynamic.3b0019 zookeeper.conf.dynamic.3b0019: server.1=100.80.2.1:2888:3888:participant;0.0.0.0:2281 server.2=100.80.2.2:2888:3888:participant;0.0.0.0:2281 server.3=100.80.2.3:2888:3888:participant;0.0.0.0:2281 logs: 2022-11-18 18:07:26,990 [main] INFO server.NettyServerCnxnFactory - bound to port 2281 2022-11-18 18:07:26,990 [main] INFO server.NettyServerCnxnFactory - binding to port 0.0.0.0/0.0.0.0:2281 2022-11-18 18:07:26,995 [main] ERROR quorum.QuorumPeerMain - Unexpected exception, exiting abnormally java.net.BindException: Address already in use at java.base/sun.nio.ch.Net.bind0(Native Method) at java.base/sun.nio.ch.Net.bind(Net.java:459) at java.base/sun.nio.ch.Net.bind(Net.java:448) at java.base/sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:227) at io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:141) at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:562) at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:506) at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:491) at io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:973) at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:260) at io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:356) at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:503) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:829) So, it seems like it's trying to bind twice to port 2281 using Netty. If I remove "secureClientPort" from zookeeper.conf and restart, then ZooKeeper starts up, but as you can see in the following logs connections exception out (presumably because TLS is disabled): 2022-11-18 18:23:57,851 [nioEventLoopGroup-4-1] WARN server.NettyServerCnxn - Closing connection to /100.80.2.5:36609 java.io.IOException: Len error 369296129 at org.apache.zookeeper.server.NettyServerCnxn.receiveMessage(NettyServerCnxn.java:521) at org.apache.zookeeper.server.NettyServerCnxn.processMessage(NettyServerCnxn.java:374) at org.apache.zookeeper.server.NettyServerCnxnFactory$CnxnChannelHandler.channelRead(NettyServerCnxnFactory.java:357) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) /Ryan On 11/17/22, 8:16 AM, "Enrico Olivelli"
Re: Confusion around client ports and dynamic reconfiguration
Ruel, Il giorno mer 16 nov 2022 alle ore 16:15 Ruel, Ryan ha scritto: > > It seems that specifying the SECURE client port in the reconfig command does > work, while also keeping the same port defined as "secureClientPort" in > zookeeper.conf. > > (I thought I had tried this, but may have missed this combination) > > In any case, some clarification within the documentation may be helpful! Would you like to send a PR to add these clarifications? Thanks Enrico > > /Ryan > > On 11/15/22, 10:10 AM, "Ruel, Ryan" wrote: > > In my ZooKeeper setup, I am strictly using TLS for both client and quorum > communication. > > In zookeeper.conf, I have “secureClientPort=2281” defined, and do not > have any “clientPort” option set. > > In the 3.8.0 documentation on dynamic reconfiguration > (https://urldefense.com/v3/__https://zookeeper.apache.org/doc/r3.8.0/zookeeperReconfig.html__;!!GjvTz_vk!T5lRlM3A1syL82ZMBR5kWiVdxaCcKIlRhDY-6muCLvhez9gyJTYbbhMkWMiee0evVrX0MQc-eYwvZhl1$ > ), the documentation says that the old “clientPort” configuration option > should not be specified, and instead the new server keyword specification > should look like this: > > server. = ::[:role];[ address>:]** > > However, this specification doesn’t consider the secure client port from > what I can tell. > > In some cases where the server keyword is used, I can just eliminate > putting in the client port address and client port, such as within the quorum > peer configuration (in zookeeper.conf or within the dynamic configuration > file). > > In other cases, however, such as using the “reconfig” command in the ZK > cli utility, the client port MUST be specified, or a “bad argument” type > error is produced. > > I of course don’t want to put a dummy port number in the server > specification which would then enable insecure communication. > > What’s the recommendation for using secure communication only while also > using dynamic reconfiguration? > > P.S. Another interesting bit in the documentation is the example: > server.1=125.23.63.23:2780:2783:participant;2791 > server.2=125.23.63.24:2781:2784:participant;2792 > server.3=125.23.63.25:2782:2785:participant;2793 > > In what use case would you want to use entirely different ports for each > server? Or is this just a demonstration that this is possible? > > /Ryan > > > > > >
Re: Confusion around client ports and dynamic reconfiguration
It seems that specifying the SECURE client port in the reconfig command does work, while also keeping the same port defined as "secureClientPort" in zookeeper.conf. (I thought I had tried this, but may have missed this combination) In any case, some clarification within the documentation may be helpful! /Ryan On 11/15/22, 10:10 AM, "Ruel, Ryan" wrote: In my ZooKeeper setup, I am strictly using TLS for both client and quorum communication. In zookeeper.conf, I have “secureClientPort=2281” defined, and do not have any “clientPort” option set. In the 3.8.0 documentation on dynamic reconfiguration (https://urldefense.com/v3/__https://zookeeper.apache.org/doc/r3.8.0/zookeeperReconfig.html__;!!GjvTz_vk!T5lRlM3A1syL82ZMBR5kWiVdxaCcKIlRhDY-6muCLvhez9gyJTYbbhMkWMiee0evVrX0MQc-eYwvZhl1$ ), the documentation says that the old “clientPort” configuration option should not be specified, and instead the new server keyword specification should look like this: server. = ::[:role];[:]** However, this specification doesn’t consider the secure client port from what I can tell. In some cases where the server keyword is used, I can just eliminate putting in the client port address and client port, such as within the quorum peer configuration (in zookeeper.conf or within the dynamic configuration file). In other cases, however, such as using the “reconfig” command in the ZK cli utility, the client port MUST be specified, or a “bad argument” type error is produced. I of course don’t want to put a dummy port number in the server specification which would then enable insecure communication. What’s the recommendation for using secure communication only while also using dynamic reconfiguration? P.S. Another interesting bit in the documentation is the example: server.1=125.23.63.23:2780:2783:participant;2791 server.2=125.23.63.24:2781:2784:participant;2792 server.3=125.23.63.25:2782:2785:participant;2793 In what use case would you want to use entirely different ports for each server? Or is this just a demonstration that this is possible? /Ryan
Confusion around client ports and dynamic reconfiguration
In my ZooKeeper setup, I am strictly using TLS for both client and quorum communication. In zookeeper.conf, I have “secureClientPort=2281” defined, and do not have any “clientPort” option set. In the 3.8.0 documentation on dynamic reconfiguration (https://zookeeper.apache.org/doc/r3.8.0/zookeeperReconfig.html), the documentation says that the old “clientPort” configuration option should not be specified, and instead the new server keyword specification should look like this: server. = ::[:role];[:]** However, this specification doesn’t consider the secure client port from what I can tell. In some cases where the server keyword is used, I can just eliminate putting in the client port address and client port, such as within the quorum peer configuration (in zookeeper.conf or within the dynamic configuration file). In other cases, however, such as using the “reconfig” command in the ZK cli utility, the client port MUST be specified, or a “bad argument” type error is produced. I of course don’t want to put a dummy port number in the server specification which would then enable insecure communication. What’s the recommendation for using secure communication only while also using dynamic reconfiguration? P.S. Another interesting bit in the documentation is the example: server.1=125.23.63.23:2780:2783:participant;2791 server.2=125.23.63.24:2781:2784:participant;2792 server.3=125.23.63.25:2782:2785:participant;2793 In what use case would you want to use entirely different ports for each server? Or is this just a demonstration that this is possible? /Ryan
