Re: upgrade from 3.4.5 to 3.5.6
Hi, Yes, all the configurations are the same on all 5 nodes. I have checked more than 10 times. Before scaling , 3 node cluster was working fine, but scaling from 3 to 5 node, then we are facing this problem. only question is on 3 node cluster it is working fine, but after scaling it is not working and zkcli is not able to connect. but one case I have seen. 1. if i remove all the SSL configuration from all the nodes and then applying again all SSL property then it is working fine. 2 cases 1st case :- 3 node cluster witout SSL is working fine scaled the cluster till 5 node without SSL is also working fine after that when applying SSL on all 5 nodes then it is also working 2nd case :- 3 node cluster with SSL is working fine scaled the cluster till 5 node with SSL is not working in both cases the configuration area same and certificate is also same. Thanks, - Kuldeep Singh Budania On Wed, Oct 14, 2020 at 5:41 PM Szalay-Bekő Máté wrote: > the config looks OK in general... > > - are you sure the same configs are used on all ZK servers? > - does the truststores accept all keys on the keystores? (if the > truststores of the old servers had to be modified, then did you restart the > old servers with the updated truststores?) > - did the 3 node ZK cluster work with SSL? (were you able to connect to it > with the client using SSL?) > > also: do you really need client authentication with SSL? (I see you are > using SASL too) > If you only need SSL for wire encryption, then you can try > with ssl.clientAuth=none (see the admin guide). Although that feature was > broken on 3.5.6, got fixed on 3.5.7 according to the doc. > > best regards, > Mate > > On Wed, Oct 14, 2020 at 1:10 PM kuldeep singh > wrote: > > > Sorry, > > secureClientPort=2182 > > > > Thanks, > > - > > Kuldeep Singh Budania > > > > > > > > On Wed, Oct 14, 2020 at 4:18 PM kuldeep singh > > > wrote: > > > > > Thanks for reply > > > > > > zoo.cfg > > > --- > > > secureClientPort=2181 > > > serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory > > > initLimit=10 > > > syncLimit=5 > > > dataDir= data directory (not mentioning exact path here) > > > tickTime=2000 > > > autopurge.snapRetainCount=3 > > > autopurge.purgeInterval=1 > > > admin.enableServer=false > > > standaloneEnabled=false > > > jute.maxbuffer=2147483648 > > > server.1=host1_priv:10288:10388 > > > server.2=host2_priv:10288:10388 > > > server.3=host3_priv:10288:10388 > > > server.4=host4_priv:10288:10388 > > > server.5=host5_priv:10288:10388 > > > quorum.auth.enableSasl=true > > > quorum.auth.learnerRequireSasl=true > > > quorum.auth.serverRequireSasl=true > > > quorum.auth.learner.loginContext=QuorumLearner > > > quorum.auth.server.loginContext=QuorumServer > > > quorum.cnxn.threads.size=10 > > > - > > > java.env > > > > > > export > > > > > > SERVER_JVMFLAGS="-Dzookeeper.serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory > > > -Dzookeeper.ssl.keyStore.location=keystore.jks > > > -Dzookeeper.ssl.keyStore.password= > > > -Dzookeeper.ssl.trustStore.location= keystore.jks > > > -Dzookeeper.ssl.trustStore.password= > > > -Djava.security.auth.login.config=zookeeper-jaas.conf" > > > > > > export > > > > > > CLIENT_JVMFLAGS="-Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty > > > -Dzookeeper.client.secure=true -Dzookeeper.ssl.keyStore.location= > > > keystore.jks -Dzookeeper.ssl.keyStore.password= > > > -Dzookeeper.ssl.trustStore.location=keystore.jks > > > -Dzookeeper.ssl.trustStore.password= > > > -Dzookeeper.ssl.hostnameVerification=false" > > > > > > Thanks, > > > - > > > Kuldeep Singh Budania > > > > > > > > > > > > On Wed, Oct 14, 2020 at 4:12 PM Szalay-Bekő Máté < > > > szalay.beko.m...@gmail.com> wrote: > > > > > >> These log messages indicate that a client (or an other ZooKeeper > server) > > >> is > > >> trying to connect without SSL to a ZooKeeper process that expects SSL. > > >> I assume this will be a configuration issue then. > > >> > > >> Best regards, > > >> Mate > > >> > > >> On Wed, Oct 14, 2020 at 12:30 PM kuldeep singh < > > kuldeep.sing...@gmail.com > > >> > > > >> wrote: > > >> > > >> > Hi, > > >> > > > >> > more logs > > >> > > > >> > 2020-10-14 12:25:05,106 - ERROR > > >> > > > >> > > > >> > > > [nioEventLoopGroup-7-4:NettyServerCnxnFactory$CnxnChannelHandler$CertificateVerifier@257 > > >> > ] > > >> > - Unsuccessful handshake with session 0x0 > > >> > > > >> > 2020-10-14 12:25:05,107 - WARN > > >> > [nioEventLoopGroup-7-4:NettyServerCnxnFactory$CnxnChannelHandler@138 > ] > > - > > >> > Exception caught > > >> > > > >> > io.netty.handler.codec.DecoderException: > > >> > io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: > > >> > 737276720a > > >> > > > >> > at > > >> > > > >> > > > >> > > >
Re: upgrade from 3.4.5 to 3.5.6
the config looks OK in general... - are you sure the same configs are used on all ZK servers? - does the truststores accept all keys on the keystores? (if the truststores of the old servers had to be modified, then did you restart the old servers with the updated truststores?) - did the 3 node ZK cluster work with SSL? (were you able to connect to it with the client using SSL?) also: do you really need client authentication with SSL? (I see you are using SASL too) If you only need SSL for wire encryption, then you can try with ssl.clientAuth=none (see the admin guide). Although that feature was broken on 3.5.6, got fixed on 3.5.7 according to the doc. best regards, Mate On Wed, Oct 14, 2020 at 1:10 PM kuldeep singh wrote: > Sorry, > secureClientPort=2182 > > Thanks, > - > Kuldeep Singh Budania > > > > On Wed, Oct 14, 2020 at 4:18 PM kuldeep singh > wrote: > > > Thanks for reply > > > > zoo.cfg > > --- > > secureClientPort=2181 > > serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory > > initLimit=10 > > syncLimit=5 > > dataDir= data directory (not mentioning exact path here) > > tickTime=2000 > > autopurge.snapRetainCount=3 > > autopurge.purgeInterval=1 > > admin.enableServer=false > > standaloneEnabled=false > > jute.maxbuffer=2147483648 > > server.1=host1_priv:10288:10388 > > server.2=host2_priv:10288:10388 > > server.3=host3_priv:10288:10388 > > server.4=host4_priv:10288:10388 > > server.5=host5_priv:10288:10388 > > quorum.auth.enableSasl=true > > quorum.auth.learnerRequireSasl=true > > quorum.auth.serverRequireSasl=true > > quorum.auth.learner.loginContext=QuorumLearner > > quorum.auth.server.loginContext=QuorumServer > > quorum.cnxn.threads.size=10 > > - > > java.env > > > > export > > > SERVER_JVMFLAGS="-Dzookeeper.serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory > > -Dzookeeper.ssl.keyStore.location=keystore.jks > > -Dzookeeper.ssl.keyStore.password= > > -Dzookeeper.ssl.trustStore.location= keystore.jks > > -Dzookeeper.ssl.trustStore.password= > > -Djava.security.auth.login.config=zookeeper-jaas.conf" > > > > export > > > CLIENT_JVMFLAGS="-Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty > > -Dzookeeper.client.secure=true -Dzookeeper.ssl.keyStore.location= > > keystore.jks -Dzookeeper.ssl.keyStore.password= > > -Dzookeeper.ssl.trustStore.location=keystore.jks > > -Dzookeeper.ssl.trustStore.password= > > -Dzookeeper.ssl.hostnameVerification=false" > > > > Thanks, > > - > > Kuldeep Singh Budania > > > > > > > > On Wed, Oct 14, 2020 at 4:12 PM Szalay-Bekő Máté < > > szalay.beko.m...@gmail.com> wrote: > > > >> These log messages indicate that a client (or an other ZooKeeper server) > >> is > >> trying to connect without SSL to a ZooKeeper process that expects SSL. > >> I assume this will be a configuration issue then. > >> > >> Best regards, > >> Mate > >> > >> On Wed, Oct 14, 2020 at 12:30 PM kuldeep singh < > kuldeep.sing...@gmail.com > >> > > >> wrote: > >> > >> > Hi, > >> > > >> > more logs > >> > > >> > 2020-10-14 12:25:05,106 - ERROR > >> > > >> > > >> > [nioEventLoopGroup-7-4:NettyServerCnxnFactory$CnxnChannelHandler$CertificateVerifier@257 > >> > ] > >> > - Unsuccessful handshake with session 0x0 > >> > > >> > 2020-10-14 12:25:05,107 - WARN > >> > [nioEventLoopGroup-7-4:NettyServerCnxnFactory$CnxnChannelHandler@138] > - > >> > Exception caught > >> > > >> > io.netty.handler.codec.DecoderException: > >> > io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: > >> > 737276720a > >> > > >> > at > >> > > >> > > >> > io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:475) > >> > > >> > at > >> > > >> > > >> > io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:283) > >> > > >> > at > >> > > >> > > >> > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) > >> > > >> > at > >> > > >> > > >> > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) > >> > > >> > at > >> > > >> > > >> > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) > >> > > >> > at > >> > > >> > > >> > io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422) > >> > > >> > at > >> > > >> > > >> > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) > >> > > >> > at > >> > > >> > > >> > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) > >> > > >> > at > >> > > >> > > >> > io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931) > >> > > >> > at > >> > > >> > > >> >
Re: upgrade from 3.4.5 to 3.5.6
Sorry, secureClientPort=2182 Thanks, - Kuldeep Singh Budania On Wed, Oct 14, 2020 at 4:18 PM kuldeep singh wrote: > Thanks for reply > > zoo.cfg > --- > secureClientPort=2181 > serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory > initLimit=10 > syncLimit=5 > dataDir= data directory (not mentioning exact path here) > tickTime=2000 > autopurge.snapRetainCount=3 > autopurge.purgeInterval=1 > admin.enableServer=false > standaloneEnabled=false > jute.maxbuffer=2147483648 > server.1=host1_priv:10288:10388 > server.2=host2_priv:10288:10388 > server.3=host3_priv:10288:10388 > server.4=host4_priv:10288:10388 > server.5=host5_priv:10288:10388 > quorum.auth.enableSasl=true > quorum.auth.learnerRequireSasl=true > quorum.auth.serverRequireSasl=true > quorum.auth.learner.loginContext=QuorumLearner > quorum.auth.server.loginContext=QuorumServer > quorum.cnxn.threads.size=10 > - > java.env > > export > SERVER_JVMFLAGS="-Dzookeeper.serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory > -Dzookeeper.ssl.keyStore.location=keystore.jks > -Dzookeeper.ssl.keyStore.password= > -Dzookeeper.ssl.trustStore.location= keystore.jks > -Dzookeeper.ssl.trustStore.password= > -Djava.security.auth.login.config=zookeeper-jaas.conf" > > export > CLIENT_JVMFLAGS="-Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty > -Dzookeeper.client.secure=true -Dzookeeper.ssl.keyStore.location= > keystore.jks -Dzookeeper.ssl.keyStore.password= > -Dzookeeper.ssl.trustStore.location=keystore.jks > -Dzookeeper.ssl.trustStore.password= > -Dzookeeper.ssl.hostnameVerification=false" > > Thanks, > - > Kuldeep Singh Budania > > > > On Wed, Oct 14, 2020 at 4:12 PM Szalay-Bekő Máté < > szalay.beko.m...@gmail.com> wrote: > >> These log messages indicate that a client (or an other ZooKeeper server) >> is >> trying to connect without SSL to a ZooKeeper process that expects SSL. >> I assume this will be a configuration issue then. >> >> Best regards, >> Mate >> >> On Wed, Oct 14, 2020 at 12:30 PM kuldeep singh > > >> wrote: >> >> > Hi, >> > >> > more logs >> > >> > 2020-10-14 12:25:05,106 - ERROR >> > >> > >> [nioEventLoopGroup-7-4:NettyServerCnxnFactory$CnxnChannelHandler$CertificateVerifier@257 >> > ] >> > - Unsuccessful handshake with session 0x0 >> > >> > 2020-10-14 12:25:05,107 - WARN >> > [nioEventLoopGroup-7-4:NettyServerCnxnFactory$CnxnChannelHandler@138] - >> > Exception caught >> > >> > io.netty.handler.codec.DecoderException: >> > io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: >> > 737276720a >> > >> > at >> > >> > >> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:475) >> > >> > at >> > >> > >> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:283) >> > >> > at >> > >> > >> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) >> > >> > at >> > >> > >> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) >> > >> > at >> > >> > >> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) >> > >> > at >> > >> > >> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422) >> > >> > at >> > >> > >> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) >> > >> > at >> > >> > >> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) >> > >> > at >> > >> > >> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931) >> > >> > at >> > >> > >> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) >> > >> > at >> > >> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700) >> > >> > at >> > >> > >> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635) >> > >> > at >> > >> > >> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552) >> > >> > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514) >> > >> > at >> > >> > >> io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044) >> > >> > at >> > >> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) >> > >> > at >> > >> > >> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) >> > >> > at java.lang.Thread.run(Thread.java:748) >> > >> > Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS >> > record: 737276720a >> > >> > at >> > >>
Re: upgrade from 3.4.5 to 3.5.6
Thanks for reply zoo.cfg --- secureClientPort=2181 serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory initLimit=10 syncLimit=5 dataDir= data directory (not mentioning exact path here) tickTime=2000 autopurge.snapRetainCount=3 autopurge.purgeInterval=1 admin.enableServer=false standaloneEnabled=false jute.maxbuffer=2147483648 server.1=host1_priv:10288:10388 server.2=host2_priv:10288:10388 server.3=host3_priv:10288:10388 server.4=host4_priv:10288:10388 server.5=host5_priv:10288:10388 quorum.auth.enableSasl=true quorum.auth.learnerRequireSasl=true quorum.auth.serverRequireSasl=true quorum.auth.learner.loginContext=QuorumLearner quorum.auth.server.loginContext=QuorumServer quorum.cnxn.threads.size=10 - java.env export SERVER_JVMFLAGS="-Dzookeeper.serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory -Dzookeeper.ssl.keyStore.location=keystore.jks -Dzookeeper.ssl.keyStore.password= -Dzookeeper.ssl.trustStore.location= keystore.jks -Dzookeeper.ssl.trustStore.password= -Djava.security.auth.login.config=zookeeper-jaas.conf" export CLIENT_JVMFLAGS="-Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty -Dzookeeper.client.secure=true -Dzookeeper.ssl.keyStore.location= keystore.jks -Dzookeeper.ssl.keyStore.password= -Dzookeeper.ssl.trustStore.location=keystore.jks -Dzookeeper.ssl.trustStore.password= -Dzookeeper.ssl.hostnameVerification=false" Thanks, - Kuldeep Singh Budania On Wed, Oct 14, 2020 at 4:12 PM Szalay-Bekő Máté wrote: > These log messages indicate that a client (or an other ZooKeeper server) is > trying to connect without SSL to a ZooKeeper process that expects SSL. > I assume this will be a configuration issue then. > > Best regards, > Mate > > On Wed, Oct 14, 2020 at 12:30 PM kuldeep singh > wrote: > > > Hi, > > > > more logs > > > > 2020-10-14 12:25:05,106 - ERROR > > > > > [nioEventLoopGroup-7-4:NettyServerCnxnFactory$CnxnChannelHandler$CertificateVerifier@257 > > ] > > - Unsuccessful handshake with session 0x0 > > > > 2020-10-14 12:25:05,107 - WARN > > [nioEventLoopGroup-7-4:NettyServerCnxnFactory$CnxnChannelHandler@138] - > > Exception caught > > > > io.netty.handler.codec.DecoderException: > > io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: > > 737276720a > > > > at > > > > > io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:475) > > > > at > > > > > io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:283) > > > > at > > > > > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) > > > > at > > > > > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) > > > > at > > > > > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) > > > > at > > > > > io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422) > > > > at > > > > > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) > > > > at > > > > > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) > > > > at > > > > > io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931) > > > > at > > > > > io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) > > > > at > > > io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700) > > > > at > > > > > io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635) > > > > at > > > > > io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552) > > > > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514) > > > > at > > > > > io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044) > > > > at > > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > > > > at > > > > > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > > > > at java.lang.Thread.run(Thread.java:748) > > > > Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS > > record: 737276720a > > > > at > > io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1203) > > > > at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1271) > > > > at > > > > > io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:505) > > > > at > > > > > io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:444) > > > > ... 17 more > > > >
Re: upgrade from 3.4.5 to 3.5.6
These log messages indicate that a client (or an other ZooKeeper server) is trying to connect without SSL to a ZooKeeper process that expects SSL. I assume this will be a configuration issue then. Best regards, Mate On Wed, Oct 14, 2020 at 12:30 PM kuldeep singh wrote: > Hi, > > more logs > > 2020-10-14 12:25:05,106 - ERROR > > [nioEventLoopGroup-7-4:NettyServerCnxnFactory$CnxnChannelHandler$CertificateVerifier@257 > ] > - Unsuccessful handshake with session 0x0 > > 2020-10-14 12:25:05,107 - WARN > [nioEventLoopGroup-7-4:NettyServerCnxnFactory$CnxnChannelHandler@138] - > Exception caught > > io.netty.handler.codec.DecoderException: > io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: > 737276720a > > at > > io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:475) > > at > > io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:283) > > at > > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) > > at > > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) > > at > > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) > > at > > io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422) > > at > > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) > > at > > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) > > at > > io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931) > > at > > io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) > > at > io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700) > > at > > io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635) > > at > > io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552) > > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514) > > at > > io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044) > > at > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > > at > > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > > at java.lang.Thread.run(Thread.java:748) > > Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS > record: 737276720a > > at > io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1203) > > at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1271) > > at > > io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:505) > > at > > io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:444) > > ... 17 more > > Thanks, > - > Kuldeep Singh Budania > Software Architect > > > > On Wed, Oct 14, 2020 at 3:51 PM kuldeep singh > wrote: > > > Hi, > > > > > > below error is coming on zookeeper logs on SSL. > > > > 2020-10-14 12:18:27,410 - WARN > [nioEventLoopGroup-4-16:NettyServerCnxn@540] > > - Closing connection to /127.0.0.1:49470 > > java.io.IOException: Len error 369296128 > > at > > > org.apache.zookeeper.server.NettyServerCnxn.receiveMessage(NettyServerCnxn.java:533) > > at > > > org.apache.zookeeper.server.NettyServerCnxn.processMessage(NettyServerCnxn.java:360) > > at > > > org.apache.zookeeper.server.NettyServerCnxnFactory$CnxnChannelHandler.channelRead(NettyServerCnxnFactory.java:184) > > at > > > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) > > at > > > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) > > at > > > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) > > at > > > io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422) > > at > > > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) > > at > > > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) > > at > > > io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931) > > at > > > io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) > > at > > > io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700) > > at > > >
Re: upgrade from 3.4.5 to 3.5.6
Hi, more logs 2020-10-14 12:25:05,106 - ERROR [nioEventLoopGroup-7-4:NettyServerCnxnFactory$CnxnChannelHandler$CertificateVerifier@257] - Unsuccessful handshake with session 0x0 2020-10-14 12:25:05,107 - WARN [nioEventLoopGroup-7-4:NettyServerCnxnFactory$CnxnChannelHandler@138] - Exception caught io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 737276720a at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:475) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:283) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514) at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 737276720a at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1203) at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1271) at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:505) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:444) ... 17 more Thanks, - Kuldeep Singh Budania Software Architect On Wed, Oct 14, 2020 at 3:51 PM kuldeep singh wrote: > Hi, > > > below error is coming on zookeeper logs on SSL. > > 2020-10-14 12:18:27,410 - WARN [nioEventLoopGroup-4-16:NettyServerCnxn@540] > - Closing connection to /127.0.0.1:49470 > java.io.IOException: Len error 369296128 > at > org.apache.zookeeper.server.NettyServerCnxn.receiveMessage(NettyServerCnxn.java:533) > at > org.apache.zookeeper.server.NettyServerCnxn.processMessage(NettyServerCnxn.java:360) > at > org.apache.zookeeper.server.NettyServerCnxnFactory$CnxnChannelHandler.channelRead(NettyServerCnxnFactory.java:184) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) > at > io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) > at > io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931) > at > io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) > at > io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700) > at > io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635) > at > io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552) > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514) > at > io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044) > at > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > at
Re: upgrade from 3.4.5 to 3.5.6
Hi, below error is coming on zookeeper logs on SSL. 2020-10-14 12:18:27,410 - WARN [nioEventLoopGroup-4-16:NettyServerCnxn@540] - Closing connection to /127.0.0.1:49470 java.io.IOException: Len error 369296128 at org.apache.zookeeper.server.NettyServerCnxn.receiveMessage(NettyServerCnxn.java:533) at org.apache.zookeeper.server.NettyServerCnxn.processMessage(NettyServerCnxn.java:360) at org.apache.zookeeper.server.NettyServerCnxnFactory$CnxnChannelHandler.channelRead(NettyServerCnxnFactory.java:184) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514) at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) Thanks, - Kuldeep Singh Budania On Thu, Oct 8, 2020 at 6:33 PM Szalay-Bekő Máté wrote: > Sounds like a bug or a configuration issue... > can you share the configs (before and after the scale-up) and the logs? > also: does the truststores recognise all the keys used on all the 5 > nodes? (e.g. the truststores on the old nodes accept the new keys?) > > Best Regards, > Mate > > On Thu, Oct 8, 2020 at 2:31 PM kuldeep singh > wrote: > > > > Hi, > > > > Yes, My client and server both are using certificate and have added in ZK > > and client as well. > > > > Thanks, > > - > > Kuldeep Singh Budania > > > > > > > > On Thu, Oct 8, 2020 at 5:56 PM Enrico Olivelli > wrote: > > > > > Il giorno gio 8 ott 2020 alle ore 14:17 kuldeep singh < > > > kuldeep.sing...@gmail.com> ha scritto: > > > > > > > Hi Team, > > > > > > > > I am facing one issue in SSL communication between client and > zookeeper > > > > server. > > > > > > > > ZK 3.5.6 version > > > > > > > > 1. Mi on 3 node > > > > 2. Applying SSL and 3 nodes cluster is working fine > > > > 3. Scaled my cluster with 2 nodes and now my cluster have 5 nodes > over > > > SSL > > > > > > > > but after scaling my SSL is not working between client and ZK server > and > > > > even not able to login using zkCli as well. > > > > > > > > Can someone provide the details please why it is happening? > > > > > > > > > > Is your client configured to use SSL ? > > > > > > Enrico > > > > > > > > > > > > > > > > > Thanks, > > > > - > > > > Kuldeep Singh Budania > > > > Software Architect > > > > > > > > > > > > > > > > On Mon, Jul 13, 2020 at 2:19 PM Enrico Olivelli - Diennea > > > > wrote: > > > > > > > > > It looks like we ported it to 3.5. > > > > > > > > > > See the subtask > > > > > https://issues.apache.org/jira/browse/ZOOKEEPER-2792 > > > > > > > > > > Enrico > > > > > > > > > > Il giorno 13/07/20, 10:37 "kuldeep singh" < > kuldeep.sing...@gmail.com> > > > > ha > > > > > scritto: > > > > > > > > > > Hi Team, > > > > > > > > > > I appreciate it if I will get a response as soon as possible, > as I > > > am > > > > > stuck > > > > > at this point. > > > > > > > > > > Thanks, > > > > > - > > > > > Kuldeep Singh Budania > > > > > > > > > > > > > > > > > > > > On Mon, Jul 13, 2020 at 11:10 AM kuldeep singh < > > > > > kuldeep.sing...@gmail.com> > > > > > wrote: > > > > > > > > > > > Hi Team, > > > > > > > > > > > > Server to Server communication is not supported in 3.5.6 > version > > > > as > > > > > per > > > > > > below JIRA issue? > > > > > > > > > > > > https://issues.apache.org/jira/browse/ZOOKEEPER-2639 > > > > > > > > > > > > Thanks, > > > > > > - > > > > > > Kuldeep Singh Budania > > > > > > > > > > > > > > > > > > > > >
