Re: Zookeeper unable to expose metrics after configuring TLS
Not sure how to explain it more clearly. You cannot communicate with plain text on a TLS port and vica versa: TLS comm. cannot be accepted on plaintext ports. It’s a general thing, not just ZooKeeper. Andor > On 2020. Mar 31., at 23:57, karthick rn wrote: > > Hi Andor, > > I've tried the openssl command you shared but unable to get it working, may > be something to do with converting to PEM format the keys and certs. I'll > look into this. > >> There’s no point trying non-secure communication on the secure port as > it’s currently not unified. > I'm not getting, please can you explain it? > > Thanks, > Karthick > > > > > > > > > > > > On Tue, 31 Mar 2020 at 15:50, Patrick Hunt wrote: > >> We (Karthick can you? :-) ) should add this to the docs. PR would be great! >> https://cwiki.apache.org/confluence/display/ZOOKEEPER/HowToContribute >> >> Thanks, >> >> Patrick >> >> On Tue, Mar 31, 2020 at 7:17 AM Andor Molnar wrote: >> >>> Hi Karthick, >>> >>> The following command works for me on the secure port (1181): >>> >>> (echo "srvr"; sleep 1) | openssl s_client -connect zkhost:1181 -cert >>> cert.pem -key ./key.pem >>> >>> I had to add sleep, because openssl client closes the connection as soon >>> as stdin ends. >>> There’s no point trying non-secure communication on the secure port as >>> it’s currently not unified. >>> >>> Andor >>> >>> >>> >>> >>> On 2020. Mar 31., at 15:22, karthick rn >>> wrote: Thanks Enrico for sharing the jira. This is great! With the below config, I'm now able to run the 4LW commands >> successfully, also the downstream systems that was relying on the 4LW commands >> started displaying the metrics. Thanks for your help. #secureClientPort=2281 clientPort=2281 client.portUnification=True - Karthick On Mon, 30 Mar 2020 at 21:59, Enrico Olivelli >>> wrote: > You may be interested in Port unification, contributed by Facebook: > > https://issues.apache.org/jira/browse/ZOOKEEPER-3388 > https://issues.apache.org/jira/browse/ZOOKEEPER-3371 > > Enrico > > Il giorno lun 30 mar 2020 alle ore 13:33 karthick rn > ha scritto: >> >> Hi Mate, >> >> Thanks for suggesting these options in detail >> >> 1) We are already using AdminServer as an alternate to the 4LW, >>> hopefully >> we'll look at modifying the downstream systems to use REST instead of >>> the >> 4LW commands. >> >> 2) Added "clientPort=2181" back to the configs and tested "srvr" & >>> other >> whitelisted 4LW commands and they all work now :) >> >> 3) When I configure the same port "2281" for both secure and unsecure >> communication with "client.portUnification=true", the JVM exits with >>> Bind >> exception stating the "Address already in use" & unable to start ZK. >> >> For short term, I think we'd run a mixed-mode communication like you >> mentioned in option 2 & whitelist only specific 4LW commands required >>> and >> not all. >> >> Appreciate if someone can confirm if the 4LW is expected to work >>> against >> secure client port or not so we can update the doc accordingly. >> Thanks >> again! >> >> Regards, >> Karthick >> >> On Mon, 30 Mar 2020 at 09:30, Szalay-Bekő Máté < > szalay.beko.m...@gmail.com> >> wrote: >> >>> Hi Karthick, >>> >>> I am not sure if "echo srvr | nc localhost 2281" is expected to work >>> against the secure client port. I don't think so, but maybe others >>> know >>> better. I think you have the following options: >>> >>> 1) use the admin server which is a HTTP interface where the 4LW > commands >>> are available on a REST protocol (see >>> > >>> >> https://zookeeper.apache.org/doc/r3.6.0/zookeeperAdmin.html#sc_adminserver >>> ) >>> >>> 2) if AdminServer is not an option for you, then you can configure >>> ZooKeeper to use both secure and unsecure ports. And use the >> unsecure > port >>> for 4LW commands, while use the secure port for the rest of the > traffic. >>> E.g.: >>> clientPort=2281 >>> secureClientPort=2282 >>> >>> 3) you can even configure ZooKeeper to use the same port for both >> TLS > and >>> unsecure communication. I haven't used 4LW commands with port > unification, >>> but I assume it works: >>> client.portUnification=true >>> >>> I hope some of these options will work for you. >>> >>> Kind regards, >>> Mate >>> >>> On Mon, Mar 30, 2020 at 12:24 AM karthick rn < > karthick.narend...@gmail.com >>> wrote: >>> Hello, After configuring TLS, running "echo srvr | nc localhost 2281" or >> any >>> other 4LW doesn’t show any output. The below messages are printed on
Re: Zookeeper unable to expose metrics after configuring TLS
Hi Andor, I've tried the openssl command you shared but unable to get it working, may be something to do with converting to PEM format the keys and certs. I'll look into this. > There’s no point trying non-secure communication on the secure port as it’s currently not unified. I'm not getting, please can you explain it? Thanks, Karthick On Tue, 31 Mar 2020 at 15:50, Patrick Hunt wrote: > We (Karthick can you? :-) ) should add this to the docs. PR would be great! > https://cwiki.apache.org/confluence/display/ZOOKEEPER/HowToContribute > > Thanks, > > Patrick > > On Tue, Mar 31, 2020 at 7:17 AM Andor Molnar wrote: > > > Hi Karthick, > > > > The following command works for me on the secure port (1181): > > > > (echo "srvr"; sleep 1) | openssl s_client -connect zkhost:1181 -cert > > cert.pem -key ./key.pem > > > > I had to add sleep, because openssl client closes the connection as soon > > as stdin ends. > > There’s no point trying non-secure communication on the secure port as > > it’s currently not unified. > > > > Andor > > > > > > > > > > > > > On 2020. Mar 31., at 15:22, karthick rn > > wrote: > > > > > > Thanks Enrico for sharing the jira. This is great! > > > > > > With the below config, I'm now able to run the 4LW commands > successfully, > > > also the downstream systems that was relying on the 4LW commands > started > > > displaying the metrics. Thanks for your help. > > > > > > #secureClientPort=2281 > > > > > > clientPort=2281 > > > > > > client.portUnification=True > > > > > > > > > - Karthick > > > > > > > > > > > > On Mon, 30 Mar 2020 at 21:59, Enrico Olivelli > > wrote: > > > > > >> You may be interested in Port unification, contributed by Facebook: > > >> > > >> https://issues.apache.org/jira/browse/ZOOKEEPER-3388 > > >> https://issues.apache.org/jira/browse/ZOOKEEPER-3371 > > >> > > >> Enrico > > >> > > >> Il giorno lun 30 mar 2020 alle ore 13:33 karthick rn > > >> ha scritto: > > >>> > > >>> Hi Mate, > > >>> > > >>> Thanks for suggesting these options in detail > > >>> > > >>> 1) We are already using AdminServer as an alternate to the 4LW, > > hopefully > > >>> we'll look at modifying the downstream systems to use REST instead of > > the > > >>> 4LW commands. > > >>> > > >>> 2) Added "clientPort=2181" back to the configs and tested "srvr" & > > other > > >>> whitelisted 4LW commands and they all work now :) > > >>> > > >>> 3) When I configure the same port "2281" for both secure and unsecure > > >>> communication with "client.portUnification=true", the JVM exits with > > Bind > > >>> exception stating the "Address already in use" & unable to start ZK. > > >>> > > >>> For short term, I think we'd run a mixed-mode communication like you > > >>> mentioned in option 2 & whitelist only specific 4LW commands required > > and > > >>> not all. > > >>> > > >>> Appreciate if someone can confirm if the 4LW is expected to work > > against > > >>> secure client port or not so we can update the doc accordingly. > Thanks > > >>> again! > > >>> > > >>> Regards, > > >>> Karthick > > >>> > > >>> On Mon, 30 Mar 2020 at 09:30, Szalay-Bekő Máté < > > >> szalay.beko.m...@gmail.com> > > >>> wrote: > > >>> > > Hi Karthick, > > > > I am not sure if "echo srvr | nc localhost 2281" is expected to work > > against the secure client port. I don't think so, but maybe others > > know > > better. I think you have the following options: > > > > 1) use the admin server which is a HTTP interface where the 4LW > > >> commands > > are available on a REST protocol (see > > > > >> > > > https://zookeeper.apache.org/doc/r3.6.0/zookeeperAdmin.html#sc_adminserver > > ) > > > > 2) if AdminServer is not an option for you, then you can configure > > ZooKeeper to use both secure and unsecure ports. And use the > unsecure > > >> port > > for 4LW commands, while use the secure port for the rest of the > > >> traffic. > > E.g.: > > clientPort=2281 > > secureClientPort=2282 > > > > 3) you can even configure ZooKeeper to use the same port for both > TLS > > >> and > > unsecure communication. I haven't used 4LW commands with port > > >> unification, > > but I assume it works: > > client.portUnification=true > > > > I hope some of these options will work for you. > > > > Kind regards, > > Mate > > > > On Mon, Mar 30, 2020 at 12:24 AM karthick rn < > > >> karthick.narend...@gmail.com > > > > > wrote: > > > > > Hello, > > > > > > After configuring TLS, running "echo srvr | nc localhost 2281" or > any > > other > > > 4LW doesn’t show any output. The below messages are printed on the > > >> ZK log > > > whilst running the ‘srvr’ command. Also tried adding > > > "4lw.commands.whitelist=*" to zoo.cfg but still no difference. > > >> However, > > > disabling TLS I'm able to see all 4LW working as expected. > > > > > > Let me know if
Re: Zookeeper unable to expose metrics after configuring TLS
We (Karthick can you? :-) ) should add this to the docs. PR would be great! https://cwiki.apache.org/confluence/display/ZOOKEEPER/HowToContribute Thanks, Patrick On Tue, Mar 31, 2020 at 7:17 AM Andor Molnar wrote: > Hi Karthick, > > The following command works for me on the secure port (1181): > > (echo "srvr"; sleep 1) | openssl s_client -connect zkhost:1181 -cert > cert.pem -key ./key.pem > > I had to add sleep, because openssl client closes the connection as soon > as stdin ends. > There’s no point trying non-secure communication on the secure port as > it’s currently not unified. > > Andor > > > > > > > On 2020. Mar 31., at 15:22, karthick rn > wrote: > > > > Thanks Enrico for sharing the jira. This is great! > > > > With the below config, I'm now able to run the 4LW commands successfully, > > also the downstream systems that was relying on the 4LW commands started > > displaying the metrics. Thanks for your help. > > > > #secureClientPort=2281 > > > > clientPort=2281 > > > > client.portUnification=True > > > > > > - Karthick > > > > > > > > On Mon, 30 Mar 2020 at 21:59, Enrico Olivelli > wrote: > > > >> You may be interested in Port unification, contributed by Facebook: > >> > >> https://issues.apache.org/jira/browse/ZOOKEEPER-3388 > >> https://issues.apache.org/jira/browse/ZOOKEEPER-3371 > >> > >> Enrico > >> > >> Il giorno lun 30 mar 2020 alle ore 13:33 karthick rn > >> ha scritto: > >>> > >>> Hi Mate, > >>> > >>> Thanks for suggesting these options in detail > >>> > >>> 1) We are already using AdminServer as an alternate to the 4LW, > hopefully > >>> we'll look at modifying the downstream systems to use REST instead of > the > >>> 4LW commands. > >>> > >>> 2) Added "clientPort=2181" back to the configs and tested "srvr" & > other > >>> whitelisted 4LW commands and they all work now :) > >>> > >>> 3) When I configure the same port "2281" for both secure and unsecure > >>> communication with "client.portUnification=true", the JVM exits with > Bind > >>> exception stating the "Address already in use" & unable to start ZK. > >>> > >>> For short term, I think we'd run a mixed-mode communication like you > >>> mentioned in option 2 & whitelist only specific 4LW commands required > and > >>> not all. > >>> > >>> Appreciate if someone can confirm if the 4LW is expected to work > against > >>> secure client port or not so we can update the doc accordingly. Thanks > >>> again! > >>> > >>> Regards, > >>> Karthick > >>> > >>> On Mon, 30 Mar 2020 at 09:30, Szalay-Bekő Máté < > >> szalay.beko.m...@gmail.com> > >>> wrote: > >>> > Hi Karthick, > > I am not sure if "echo srvr | nc localhost 2281" is expected to work > against the secure client port. I don't think so, but maybe others > know > better. I think you have the following options: > > 1) use the admin server which is a HTTP interface where the 4LW > >> commands > are available on a REST protocol (see > > >> > https://zookeeper.apache.org/doc/r3.6.0/zookeeperAdmin.html#sc_adminserver > ) > > 2) if AdminServer is not an option for you, then you can configure > ZooKeeper to use both secure and unsecure ports. And use the unsecure > >> port > for 4LW commands, while use the secure port for the rest of the > >> traffic. > E.g.: > clientPort=2281 > secureClientPort=2282 > > 3) you can even configure ZooKeeper to use the same port for both TLS > >> and > unsecure communication. I haven't used 4LW commands with port > >> unification, > but I assume it works: > client.portUnification=true > > I hope some of these options will work for you. > > Kind regards, > Mate > > On Mon, Mar 30, 2020 at 12:24 AM karthick rn < > >> karthick.narend...@gmail.com > > > wrote: > > > Hello, > > > > After configuring TLS, running "echo srvr | nc localhost 2281" or any > other > > 4LW doesn’t show any output. The below messages are printed on the > >> ZK log > > whilst running the ‘srvr’ command. Also tried adding > > "4lw.commands.whitelist=*" to zoo.cfg but still no difference. > >> However, > > disabling TLS I'm able to see all 4LW working as expected. > > > > Let me know if this is a known issue when TLS is enabled? I'm using > >> ZK > v3.6 > > and have seen the same behaviour with v3.5.6 & 3.5.7. > > > > I have shared my Quorum TLS configs at the bottom, in-case if you > >> want to > > check if I'm missing something. Many thanks > > > > > > zookeeper.log: > > > > > > 2020-03-29 21:09:27,079 [myid:1] - ERROR > > [nioEventLoopGroup-4-1:NettyServerCnxnFactory$CertificateVerifier@434 > ] > >> - > > Unsuccessful handshake with session 0x0 > > > > 2020-03-29 21:09:27,083 [myid:1] - WARN > > [nioEventLoopGroup-4-1:NettyServerCnxnFactory$CnxnChannelHandler@273 > ] > >> - > > Exception caught > > > >
Re: Zookeeper unable to expose metrics after configuring TLS
Hi Karthick, The following command works for me on the secure port (1181): (echo "srvr"; sleep 1) | openssl s_client -connect zkhost:1181 -cert cert.pem -key ./key.pem I had to add sleep, because openssl client closes the connection as soon as stdin ends. There’s no point trying non-secure communication on the secure port as it’s currently not unified. Andor > On 2020. Mar 31., at 15:22, karthick rn wrote: > > Thanks Enrico for sharing the jira. This is great! > > With the below config, I'm now able to run the 4LW commands successfully, > also the downstream systems that was relying on the 4LW commands started > displaying the metrics. Thanks for your help. > > #secureClientPort=2281 > > clientPort=2281 > > client.portUnification=True > > > - Karthick > > > > On Mon, 30 Mar 2020 at 21:59, Enrico Olivelli wrote: > >> You may be interested in Port unification, contributed by Facebook: >> >> https://issues.apache.org/jira/browse/ZOOKEEPER-3388 >> https://issues.apache.org/jira/browse/ZOOKEEPER-3371 >> >> Enrico >> >> Il giorno lun 30 mar 2020 alle ore 13:33 karthick rn >> ha scritto: >>> >>> Hi Mate, >>> >>> Thanks for suggesting these options in detail >>> >>> 1) We are already using AdminServer as an alternate to the 4LW, hopefully >>> we'll look at modifying the downstream systems to use REST instead of the >>> 4LW commands. >>> >>> 2) Added "clientPort=2181" back to the configs and tested "srvr" & other >>> whitelisted 4LW commands and they all work now :) >>> >>> 3) When I configure the same port "2281" for both secure and unsecure >>> communication with "client.portUnification=true", the JVM exits with Bind >>> exception stating the "Address already in use" & unable to start ZK. >>> >>> For short term, I think we'd run a mixed-mode communication like you >>> mentioned in option 2 & whitelist only specific 4LW commands required and >>> not all. >>> >>> Appreciate if someone can confirm if the 4LW is expected to work against >>> secure client port or not so we can update the doc accordingly. Thanks >>> again! >>> >>> Regards, >>> Karthick >>> >>> On Mon, 30 Mar 2020 at 09:30, Szalay-Bekő Máté < >> szalay.beko.m...@gmail.com> >>> wrote: >>> Hi Karthick, I am not sure if "echo srvr | nc localhost 2281" is expected to work against the secure client port. I don't think so, but maybe others know better. I think you have the following options: 1) use the admin server which is a HTTP interface where the 4LW >> commands are available on a REST protocol (see >> https://zookeeper.apache.org/doc/r3.6.0/zookeeperAdmin.html#sc_adminserver ) 2) if AdminServer is not an option for you, then you can configure ZooKeeper to use both secure and unsecure ports. And use the unsecure >> port for 4LW commands, while use the secure port for the rest of the >> traffic. E.g.: clientPort=2281 secureClientPort=2282 3) you can even configure ZooKeeper to use the same port for both TLS >> and unsecure communication. I haven't used 4LW commands with port >> unification, but I assume it works: client.portUnification=true I hope some of these options will work for you. Kind regards, Mate On Mon, Mar 30, 2020 at 12:24 AM karthick rn < >> karthick.narend...@gmail.com > wrote: > Hello, > > After configuring TLS, running "echo srvr | nc localhost 2281" or any other > 4LW doesn’t show any output. The below messages are printed on the >> ZK log > whilst running the ‘srvr’ command. Also tried adding > "4lw.commands.whitelist=*" to zoo.cfg but still no difference. >> However, > disabling TLS I'm able to see all 4LW working as expected. > > Let me know if this is a known issue when TLS is enabled? I'm using >> ZK v3.6 > and have seen the same behaviour with v3.5.6 & 3.5.7. > > I have shared my Quorum TLS configs at the bottom, in-case if you >> want to > check if I'm missing something. Many thanks > > > zookeeper.log: > > > 2020-03-29 21:09:27,079 [myid:1] - ERROR > [nioEventLoopGroup-4-1:NettyServerCnxnFactory$CertificateVerifier@434] >> - > Unsuccessful handshake with session 0x0 > > 2020-03-29 21:09:27,083 [myid:1] - WARN > [nioEventLoopGroup-4-1:NettyServerCnxnFactory$CnxnChannelHandler@273] >> - > Exception caught > > io.netty.handler.codec.DecoderException: > io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: > 737276720a > >at > > >> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) > >at > > >> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) > >at > > >>
Re: Zookeeper unable to expose metrics after configuring TLS
Thanks Enrico for sharing the jira. This is great! With the below config, I'm now able to run the 4LW commands successfully, also the downstream systems that was relying on the 4LW commands started displaying the metrics. Thanks for your help. #secureClientPort=2281 clientPort=2281 client.portUnification=True - Karthick On Mon, 30 Mar 2020 at 21:59, Enrico Olivelli wrote: > You may be interested in Port unification, contributed by Facebook: > > https://issues.apache.org/jira/browse/ZOOKEEPER-3388 > https://issues.apache.org/jira/browse/ZOOKEEPER-3371 > > Enrico > > Il giorno lun 30 mar 2020 alle ore 13:33 karthick rn > ha scritto: > > > > Hi Mate, > > > > Thanks for suggesting these options in detail > > > > 1) We are already using AdminServer as an alternate to the 4LW, hopefully > > we'll look at modifying the downstream systems to use REST instead of the > > 4LW commands. > > > > 2) Added "clientPort=2181" back to the configs and tested "srvr" & other > > whitelisted 4LW commands and they all work now :) > > > > 3) When I configure the same port "2281" for both secure and unsecure > > communication with "client.portUnification=true", the JVM exits with Bind > > exception stating the "Address already in use" & unable to start ZK. > > > > For short term, I think we'd run a mixed-mode communication like you > > mentioned in option 2 & whitelist only specific 4LW commands required and > > not all. > > > > Appreciate if someone can confirm if the 4LW is expected to work against > > secure client port or not so we can update the doc accordingly. Thanks > > again! > > > > Regards, > > Karthick > > > > On Mon, 30 Mar 2020 at 09:30, Szalay-Bekő Máté < > szalay.beko.m...@gmail.com> > > wrote: > > > > > Hi Karthick, > > > > > > I am not sure if "echo srvr | nc localhost 2281" is expected to work > > > against the secure client port. I don't think so, but maybe others know > > > better. I think you have the following options: > > > > > > 1) use the admin server which is a HTTP interface where the 4LW > commands > > > are available on a REST protocol (see > > > > https://zookeeper.apache.org/doc/r3.6.0/zookeeperAdmin.html#sc_adminserver > > > ) > > > > > > 2) if AdminServer is not an option for you, then you can configure > > > ZooKeeper to use both secure and unsecure ports. And use the unsecure > port > > > for 4LW commands, while use the secure port for the rest of the > traffic. > > > E.g.: > > > clientPort=2281 > > > secureClientPort=2282 > > > > > > 3) you can even configure ZooKeeper to use the same port for both TLS > and > > > unsecure communication. I haven't used 4LW commands with port > unification, > > > but I assume it works: > > > client.portUnification=true > > > > > > I hope some of these options will work for you. > > > > > > Kind regards, > > > Mate > > > > > > On Mon, Mar 30, 2020 at 12:24 AM karthick rn < > karthick.narend...@gmail.com > > > > > > > wrote: > > > > > > > Hello, > > > > > > > > After configuring TLS, running "echo srvr | nc localhost 2281" or any > > > other > > > > 4LW doesn’t show any output. The below messages are printed on the > ZK log > > > > whilst running the ‘srvr’ command. Also tried adding > > > > "4lw.commands.whitelist=*" to zoo.cfg but still no difference. > However, > > > > disabling TLS I'm able to see all 4LW working as expected. > > > > > > > > Let me know if this is a known issue when TLS is enabled? I'm using > ZK > > > v3.6 > > > > and have seen the same behaviour with v3.5.6 & 3.5.7. > > > > > > > > I have shared my Quorum TLS configs at the bottom, in-case if you > want to > > > > check if I'm missing something. Many thanks > > > > > > > > > > > > zookeeper.log: > > > > > > > > > > > > 2020-03-29 21:09:27,079 [myid:1] - ERROR > > > > [nioEventLoopGroup-4-1:NettyServerCnxnFactory$CertificateVerifier@434] > - > > > > Unsuccessful handshake with session 0x0 > > > > > > > > 2020-03-29 21:09:27,083 [myid:1] - WARN > > > > [nioEventLoopGroup-4-1:NettyServerCnxnFactory$CnxnChannelHandler@273] > - > > > > Exception caught > > > > > > > > io.netty.handler.codec.DecoderException: > > > > io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: > > > > 737276720a > > > > > > > > at > > > > > > > > > > > > io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) > > > > > > > > at > > > > > > > > > > > > io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) > > > > > > > > at > > > > > > > > > > > > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) > > > > > > > > at > > > > > > > > > > > > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) > > > > > > > > at > > > > > > > > > > > > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) > > > > > > > > at > > > > > > > > > > > >
Re: Zookeeper unable to expose metrics after configuring TLS
You may be interested in Port unification, contributed by Facebook: https://issues.apache.org/jira/browse/ZOOKEEPER-3388 https://issues.apache.org/jira/browse/ZOOKEEPER-3371 Enrico Il giorno lun 30 mar 2020 alle ore 13:33 karthick rn ha scritto: > > Hi Mate, > > Thanks for suggesting these options in detail > > 1) We are already using AdminServer as an alternate to the 4LW, hopefully > we'll look at modifying the downstream systems to use REST instead of the > 4LW commands. > > 2) Added "clientPort=2181" back to the configs and tested "srvr" & other > whitelisted 4LW commands and they all work now :) > > 3) When I configure the same port "2281" for both secure and unsecure > communication with "client.portUnification=true", the JVM exits with Bind > exception stating the "Address already in use" & unable to start ZK. > > For short term, I think we'd run a mixed-mode communication like you > mentioned in option 2 & whitelist only specific 4LW commands required and > not all. > > Appreciate if someone can confirm if the 4LW is expected to work against > secure client port or not so we can update the doc accordingly. Thanks > again! > > Regards, > Karthick > > On Mon, 30 Mar 2020 at 09:30, Szalay-Bekő Máté > wrote: > > > Hi Karthick, > > > > I am not sure if "echo srvr | nc localhost 2281" is expected to work > > against the secure client port. I don't think so, but maybe others know > > better. I think you have the following options: > > > > 1) use the admin server which is a HTTP interface where the 4LW commands > > are available on a REST protocol (see > > https://zookeeper.apache.org/doc/r3.6.0/zookeeperAdmin.html#sc_adminserver > > ) > > > > 2) if AdminServer is not an option for you, then you can configure > > ZooKeeper to use both secure and unsecure ports. And use the unsecure port > > for 4LW commands, while use the secure port for the rest of the traffic. > > E.g.: > > clientPort=2281 > > secureClientPort=2282 > > > > 3) you can even configure ZooKeeper to use the same port for both TLS and > > unsecure communication. I haven't used 4LW commands with port unification, > > but I assume it works: > > client.portUnification=true > > > > I hope some of these options will work for you. > > > > Kind regards, > > Mate > > > > On Mon, Mar 30, 2020 at 12:24 AM karthick rn > > > > wrote: > > > > > Hello, > > > > > > After configuring TLS, running "echo srvr | nc localhost 2281" or any > > other > > > 4LW doesn’t show any output. The below messages are printed on the ZK log > > > whilst running the ‘srvr’ command. Also tried adding > > > "4lw.commands.whitelist=*" to zoo.cfg but still no difference. However, > > > disabling TLS I'm able to see all 4LW working as expected. > > > > > > Let me know if this is a known issue when TLS is enabled? I'm using ZK > > v3.6 > > > and have seen the same behaviour with v3.5.6 & 3.5.7. > > > > > > I have shared my Quorum TLS configs at the bottom, in-case if you want to > > > check if I'm missing something. Many thanks > > > > > > > > > zookeeper.log: > > > > > > > > > 2020-03-29 21:09:27,079 [myid:1] - ERROR > > > [nioEventLoopGroup-4-1:NettyServerCnxnFactory$CertificateVerifier@434] - > > > Unsuccessful handshake with session 0x0 > > > > > > 2020-03-29 21:09:27,083 [myid:1] - WARN > > > [nioEventLoopGroup-4-1:NettyServerCnxnFactory$CnxnChannelHandler@273] - > > > Exception caught > > > > > > io.netty.handler.codec.DecoderException: > > > io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: > > > 737276720a > > > > > > at > > > > > > > > io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) > > > > > > at > > > > > > > > io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) > > > > > > at > > > > > > > > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) > > > > > > at > > > > > > > > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) > > > > > > at > > > > > > > > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) > > > > > > at > > > > > > > > io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) > > > > > > at > > > > > > > > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) > > > > > > at > > > > > > > > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) > > > > > > at > > > > > > > > io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) > > > > > > at > > > > > > > > io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) > > > > > > at > > > > > io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) > > > > > > at > > > > > > > >
Re: Zookeeper unable to expose metrics after configuring TLS
Hi Mate, Thanks for suggesting these options in detail 1) We are already using AdminServer as an alternate to the 4LW, hopefully we'll look at modifying the downstream systems to use REST instead of the 4LW commands. 2) Added "clientPort=2181" back to the configs and tested "srvr" & other whitelisted 4LW commands and they all work now :) 3) When I configure the same port "2281" for both secure and unsecure communication with "client.portUnification=true", the JVM exits with Bind exception stating the "Address already in use" & unable to start ZK. For short term, I think we'd run a mixed-mode communication like you mentioned in option 2 & whitelist only specific 4LW commands required and not all. Appreciate if someone can confirm if the 4LW is expected to work against secure client port or not so we can update the doc accordingly. Thanks again! Regards, Karthick On Mon, 30 Mar 2020 at 09:30, Szalay-Bekő Máté wrote: > Hi Karthick, > > I am not sure if "echo srvr | nc localhost 2281" is expected to work > against the secure client port. I don't think so, but maybe others know > better. I think you have the following options: > > 1) use the admin server which is a HTTP interface where the 4LW commands > are available on a REST protocol (see > https://zookeeper.apache.org/doc/r3.6.0/zookeeperAdmin.html#sc_adminserver > ) > > 2) if AdminServer is not an option for you, then you can configure > ZooKeeper to use both secure and unsecure ports. And use the unsecure port > for 4LW commands, while use the secure port for the rest of the traffic. > E.g.: > clientPort=2281 > secureClientPort=2282 > > 3) you can even configure ZooKeeper to use the same port for both TLS and > unsecure communication. I haven't used 4LW commands with port unification, > but I assume it works: > client.portUnification=true > > I hope some of these options will work for you. > > Kind regards, > Mate > > On Mon, Mar 30, 2020 at 12:24 AM karthick rn > > wrote: > > > Hello, > > > > After configuring TLS, running "echo srvr | nc localhost 2281" or any > other > > 4LW doesn’t show any output. The below messages are printed on the ZK log > > whilst running the ‘srvr’ command. Also tried adding > > "4lw.commands.whitelist=*" to zoo.cfg but still no difference. However, > > disabling TLS I'm able to see all 4LW working as expected. > > > > Let me know if this is a known issue when TLS is enabled? I'm using ZK > v3.6 > > and have seen the same behaviour with v3.5.6 & 3.5.7. > > > > I have shared my Quorum TLS configs at the bottom, in-case if you want to > > check if I'm missing something. Many thanks > > > > > > zookeeper.log: > > > > > > 2020-03-29 21:09:27,079 [myid:1] - ERROR > > [nioEventLoopGroup-4-1:NettyServerCnxnFactory$CertificateVerifier@434] - > > Unsuccessful handshake with session 0x0 > > > > 2020-03-29 21:09:27,083 [myid:1] - WARN > > [nioEventLoopGroup-4-1:NettyServerCnxnFactory$CnxnChannelHandler@273] - > > Exception caught > > > > io.netty.handler.codec.DecoderException: > > io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: > > 737276720a > > > > at > > > > > io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) > > > > at > > > > > io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) > > > > at > > > > > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) > > > > at > > > > > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) > > > > at > > > > > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) > > > > at > > > > > io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) > > > > at > > > > > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) > > > > at > > > > > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) > > > > at > > > > > io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) > > > > at > > > > > io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) > > > > at > > > io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) > > > > at > > > > > io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) > > > > at > > > > > io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) > > > > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) > > > > at > > > > > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) > > > > at > > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > > > > at > > > > >
Re: Zookeeper unable to expose metrics after configuring TLS
Hi Karthick, I am not sure if "echo srvr | nc localhost 2281" is expected to work against the secure client port. I don't think so, but maybe others know better. I think you have the following options: 1) use the admin server which is a HTTP interface where the 4LW commands are available on a REST protocol (see https://zookeeper.apache.org/doc/r3.6.0/zookeeperAdmin.html#sc_adminserver) 2) if AdminServer is not an option for you, then you can configure ZooKeeper to use both secure and unsecure ports. And use the unsecure port for 4LW commands, while use the secure port for the rest of the traffic. E.g.: clientPort=2281 secureClientPort=2282 3) you can even configure ZooKeeper to use the same port for both TLS and unsecure communication. I haven't used 4LW commands with port unification, but I assume it works: client.portUnification=true I hope some of these options will work for you. Kind regards, Mate On Mon, Mar 30, 2020 at 12:24 AM karthick rn wrote: > Hello, > > After configuring TLS, running "echo srvr | nc localhost 2281" or any other > 4LW doesn’t show any output. The below messages are printed on the ZK log > whilst running the ‘srvr’ command. Also tried adding > "4lw.commands.whitelist=*" to zoo.cfg but still no difference. However, > disabling TLS I'm able to see all 4LW working as expected. > > Let me know if this is a known issue when TLS is enabled? I'm using ZK v3.6 > and have seen the same behaviour with v3.5.6 & 3.5.7. > > I have shared my Quorum TLS configs at the bottom, in-case if you want to > check if I'm missing something. Many thanks > > > zookeeper.log: > > > 2020-03-29 21:09:27,079 [myid:1] - ERROR > [nioEventLoopGroup-4-1:NettyServerCnxnFactory$CertificateVerifier@434] - > Unsuccessful handshake with session 0x0 > > 2020-03-29 21:09:27,083 [myid:1] - WARN > [nioEventLoopGroup-4-1:NettyServerCnxnFactory$CnxnChannelHandler@273] - > Exception caught > > io.netty.handler.codec.DecoderException: > io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: > 737276720a > > at > > io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) > > at > > io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) > > at > > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) > > at > > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) > > at > > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) > > at > > io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) > > at > > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) > > at > > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) > > at > > io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) > > at > > io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) > > at > io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) > > at > > io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) > > at > > io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) > > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) > > at > > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) > > at > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > > at > > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > > at java.base/java.lang.Thread.run(Thread.java:834) > > Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS > record: 737276720a > > at > io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1198) > > at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1266) > > at > > io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498) > > at > > io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437) > > ... 17 more > > > > conf/zoo.cfg: > > > > # Server configuration > > secureClientPort=2281 > > serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory > > > > # Quorum configuration > > sslQuorum=true > > ssl.quorum.keyStore.location= > > ssl.quorum.keyStore.password= > > ssl.quorum.trustStore.location= > > ssl.quorum.trustStore.password= > > > > bin/zkEnv.sh > > > > > SERVER_JVMFLAGS="-Dzookeeper.serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory > \ > > -Dzookeeper.ssl.keyStore.location= \ > > -Dzookeeper.ssl.keyStore.password=\ > > -Dzookeeper.ssl.trustStore.location= \ > >
