Re: upgrade from 3.4.5 to 3.5.6
Lint.java:346) at com.sun.tools.javac.tree.JCTree$JCVariableDecl.accept(JCTree.java:864) at com.sun.source.util.TreePathScanner.scan(TreePathScanner.java:68) at com.sun.source.util.TreeScanner.scanAndReduce(TreeScanner.java:81) at com.sun.source.util.TreeScanner.scan(TreeScanner.java:91) at com.sun.source.util.TreeScanner.scanAndReduce(TreeScanner.java:99) at com.sun.source.util.TreeScanner.visitClass(TreeScanner.java:133) at com.sun.tools.doclint.DocLint$DeclScanner.visitClass(DocLint.java:360) at com.sun.tools.doclint.DocLint$DeclScanner.visitClass(DocLint.java:346) at com.sun.tools.javac.tree.JCTree$JCClassDecl.accept(JCTree.java:720) at com.sun.source.util.TreePathScanner.scan(TreePathScanner.java:68) at com.sun.source.util.TreeScanner.scan(TreeScanner.java:91) at com.sun.source.util.TreeScanner.scanAndReduce(TreeScanner.java:99) at com.sun.source.util.TreeScanner.visitCompilationUnit(TreeScanner.java:120) at com.sun.tools.doclint.DocLint$DeclScanner.visitCompilationUnit(DocLint.java:354) at com.sun.tools.doclint.DocLint$DeclScanner.visitCompilationUnit(DocLint.java:346) at com.sun.tools.javac.tree.JCTree$JCCompilationUnit.accept(JCTree.java:550) at com.sun.source.util.TreePathScanner.scan(TreePathScanner.java:68) at com.sun.tools.doclint.DocLint$3.started(DocLint.java:296) at com.sun.tools.javac.api.ClientCodeWrapper$WrappedTaskListener.started(ClientCodeWrapper.java:668) at com.sun.tools.javac.api.MultiTaskListener.started(MultiTaskListener.java:103) at com.sun.tools.javac.main.JavaCompiler.attribute(JavaCompiler.java:1240) at com.sun.tools.javac.main.JavaCompiler.compile2(JavaCompiler.java:901) at com.sun.tools.javac.main.JavaCompiler.compile(JavaCompiler.java:860) at com.sun.tools.javac.main.Main.compile(Main.java:523) ... 27 more [ERROR] [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException Can someone help me on this issue? Thanks, ----- Kuldeep Singh Budania On Sat, Apr 4, 2020 at 5:57 PM Szalay-Bekő Máté wrote: > these exceptions can mean many things... I think this can be even normal > duding rolling restart (as some connections get broken in this case) > > However, I saw cases already when exceptions like these killed receiver or > sender threads in QuorumCnxManager / Leader Election in such a way that > they were not able to recover, so the node was unable to connect to any > quorum until restart. I remember seeing this in 3.4 too. > > Do you see these exceptions in the second server (the one which you just > upgraded in step 3)? > Is this issue reproducible? > > What is the tickTime and initLimit you use? Maybe the server just require > more time to sync? > > I would need more logs to really see what happened. Can you create a Jira > issue and upload the logs and also the ZooKeeper configs? I am happy to > take a closer look. > (if you need to re-run the test to collect the logs, then enabling DEBUG > logs would be great. The INFO level logs are usually enough for these > problems, but one can never know...) > > Kind regards, > Mate > > > On Fri, Apr 3, 2020 at 10:05 AM kuldeep singh > wrote: > > > Hi Team, > > > > I have done some POC on rolling upgrade and found below result. > > > > > >1. On 1st node upgrade zookeeper . Traffic was running fine because 2 > >nodes are already on old zookeeper. > >2. On 1st node upgrade our application and didn’t find any issue > >3. On 2nd node upgrade zookeeper but got below error and zookeeper is > >not taking any requests > >4. > > > > java.io.EOFException > > > > at java.io.DataInputStream.readInt(DataInputStream.java:392) > > > > at > > > > > org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:747) > > > > 2020-03-30 14:19:55,587 - WARN > > [RecvWorker:1:QuorumCnxManager$RecvWorker@765] - Interrupting SendWorker > > > > 2020-03-30 14:19:55,588 - ERROR [LearnerHandler-/192.168.44.73:33754 > > :LearnerHandler@562] - Unexpected exception causing shutdown while sock > > still open > > > > java.io.EOFException > > > > at java.io.DataInputStream.readInt(DataInputStream.java:392) > > > > at > > org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63) > > > > at > > > > > org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(Quoru
Re: upgrade from 3.4.5 to 3.5.6
Hi Team, I have done some POC on rolling upgrade and found below result. 1. On 1st node upgrade zookeeper . Traffic was running fine because 2 nodes are already on old zookeeper. 2. On 1st node upgrade our application and didn’t find any issue 3. On 2nd node upgrade zookeeper but got below error and zookeeper is not taking any requests 4. java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:392) at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:747) 2020-03-30 14:19:55,587 - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@765] - Interrupting SendWorker 2020-03-30 14:19:55,588 - ERROR [LearnerHandler-/192.168.44.73:33754 :LearnerHandler@562] - Unexpected exception causing shutdown while sock still open java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:392) at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63) at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:83) at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:108) at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:476) 2020-03-30 14:19:55,588 - WARN [SendWorker:1:QuorumCnxManager$SendWorker@679] - Interrupted while waiting for message on queue Please let me know is this the known issue or this is different issue which is mention in Apache zookeeper documentation when upgrading from 3.4.5 to 3.5.6 Thanks, - Kuldeep Singh Budania Software Architect On Sun, Mar 29, 2020 at 9:06 AM Alexander Shraer wrote: > +1 to what Mate said (I wrote the quoted instructions). > > > > On Tue, Mar 24, 2020 at 7:03 AM Szalay-Bekő Máté < > szalay.beko.m...@gmail.com> > wrote: > > > Hi Kuldeep, > > > > I just want to provide you some background info about our documentation. > > The reason to upgrade to 3.4.6 first is to avoid the following error: > > > > > 2013-01-30 11:32:10,663 [myid:2] - WARN [localhost/127.0.0.1:2784 > > :QuorumCnxManager@349] - Invalid server id: -65536 > > > > This error comes because of the protocol changes between ZooKeeper server > > nodes during connection initiation for leader election. In ZooKeeper 3.5 > a > > protocol version was introduced (see ZOOKEEPER-107) and since that time > the > > fist long value sent in the initial message is not the server ID but the > > protocol version (-65536). In ZooKeeper 3.4.6 we made the old 3.4 > > ZooKeepers backward compatible, so they are able to parse both the old > and > > the new protocol format (see ZOOKEEPER-1633). This issue happens only > when > > you need to use old (3.4.0 - 3.4.5) and new (3.5.0+) ZooKeeper servers > > together in the same cluster. During a rolling upgrade, this is usually > the > > case to have old and new ZooKeepers present together. > > > > The fact that you haven't seen any issues might be caused by the order of > > the servers. In ZooKeeper the connection initiation between the servers > > during the leader election follows a specific rule. As far as I remember > > always the server with the larger ID 'wins the challenge', so it is > > possible, that the old server didn't need to parse any initial message > (if > > it had the largest ID) and this is why you haven't seen the issue. Also > > having 2 nodes up from the 3 nodes cluster still makes the cluster work > (so > > you should also check if all the servers are part of the quorum). > > > > I agree with Enrico and Norbert, the safest and most stable way is > upgrade > > first to 3.4.latest, then go to 3.5.latest. Still, if you don't see that > > you would hit this specific issue (e.g. no "Invalid server id" in the log > > files), and all the three servers can handle traffic, then maybe you > don't > > need to upgrade first to 3.4.latest, it is your decision. Definitely you > > should test it first, as suggested by the others. > > > > Kind regards, > > Mate > > > > On Tue, Mar 24, 2020 at 12:29 PM Norbert Kalmar > > wrote: > > > > > Hi, > > > > > > That guide is to upgrade to 3.5.0, which was an alpha version. A lot > has > > > changed for the first stable release of 3.5.5 and then a few more, even > > > rolling upgrade issues have been fixed for 3.5.6. > > > This is a more up-to-date guide: > > > https://cwiki.apache.org/confluence/display/ZOOKEEPER/Upgrade+FAQ > > > > > > If you have done your testing (with prod snapshot!), then you can skip > > 3.4 > > > latest upgrade, but keep in mind we do our recommendations
Re: upgrade from 3.4.5 to 3.5.6
Hi, Current Zookeeper version :- 3.4.5 Upgraded version:- 3.5.6 We are not going with 3.5.7. Our final decision is zookeeper version is 3.5.6 as per your reply first we need to move latest version of 3.4.x, like below 3.4.5 -> 3.4.14 -> 3.5.6 (Correct me if I am wrong here) But if We are not facing any problem that i have shared you that we have set up of 3 node cluster where 2 node are on 3.5.6 version and 1 node on 3.4.5, Everything is running fine and didn't get any issue, So what other problem we can face if we directly move to 3.5.6 Thanks, - Kuldeep Singh Budania Software Architect On Tue, Mar 24, 2020 at 3:58 PM Enrico Olivelli wrote: > Hi > You have to upgrade to latest 3.4.x Zookeeper then you will upgrade to > 3.5.7. > All should run well without issues > > > Enrico > > Il Mar 24 Mar 2020, 10:18 kuldeep singh ha > scritto: > > > Hi Team, > > > > We are upgrading zookeeper from 3.4.5 to 3.5.6. I have set up 3 node > > cluster where 2 node are on 3.5.6 version and 1 node on 3.4.5. > > > > Everything is running fine and didn't get any issue on my system. > > > > but I found something on apache site that first we need to upgrade on > > 3.4.6 than we can upgrade to 3.5.6. So is it mandatory to go on 3.4.6 > > first. > > > > *Upgrading to 3.5.0* > > > > Upgrading a running ZooKeeper ensemble to 3.5.0 should be done only after > > upgrading your ensemble to the 3.4.6 release. Note that this is only > > necessary for rolling upgrades (if you're fine with shutting down the > > system completely, you don't have to go through 3.4.6). If you attempt a > > rolling upgrade without going through 3.4.6 (for example from 3.4.5), you > > may get the following error: > > > > 2013-01-30 11:32:10,663 [myid:2] - INFO [localhost/127.0.0.1:2784 > > :QuorumCnxManager$Listener@498] - Received connection request / > > 127.0.0.1:60876 > > > > 2013-01-30 11:32:10,663 [myid:2] - WARN [localhost/127.0.0.1:2784 > > :QuorumCnxManager@349] - Invalid server id: -65536 > > > > During a rolling upgrade, each server is taken down in turn and rebooted > > with the new 3.5.0 binaries. Before starting the server with 3.5.0 > > binaries, we highly recommend updating the configuration file so that all > > server statements "server.x=..." contain client ports (see the section > > Specifying > > the client port). As explained earlier you may leave the configuration > in a > > single file, as well as leave the clientPort/clientPortAddress statements > > (although if you specify client ports in the new format, these statements > > are now redundant). > > > > Could you please let me know about this case. Appreciate if respond soon. > > > > Thanks, > > - > > Kuldeep Singh Budania > > >
upgrade from 3.4.5 to 3.5.6
Hi Team, We are upgrading zookeeper from 3.4.5 to 3.5.6. I have set up 3 node cluster where 2 node are on 3.5.6 version and 1 node on 3.4.5. Everything is running fine and didn't get any issue on my system. but I found something on apache site that first we need to upgrade on 3.4.6 than we can upgrade to 3.5.6. So is it mandatory to go on 3.4.6 first. *Upgrading to 3.5.0* Upgrading a running ZooKeeper ensemble to 3.5.0 should be done only after upgrading your ensemble to the 3.4.6 release. Note that this is only necessary for rolling upgrades (if you're fine with shutting down the system completely, you don't have to go through 3.4.6). If you attempt a rolling upgrade without going through 3.4.6 (for example from 3.4.5), you may get the following error: 2013-01-30 11:32:10,663 [myid:2] - INFO [localhost/127.0.0.1:2784 :QuorumCnxManager$Listener@498] - Received connection request / 127.0.0.1:60876 2013-01-30 11:32:10,663 [myid:2] - WARN [localhost/127.0.0.1:2784 :QuorumCnxManager@349] - Invalid server id: -65536 During a rolling upgrade, each server is taken down in turn and rebooted with the new 3.5.0 binaries. Before starting the server with 3.5.0 binaries, we highly recommend updating the configuration file so that all server statements "server.x=..." contain client ports (see the section Specifying the client port). As explained earlier you may leave the configuration in a single file, as well as leave the clientPort/clientPortAddress statements (although if you specify client ports in the new format, these statements are now redundant). Could you please let me know about this case. Appreciate if respond soon. Thanks, ----- Kuldeep Singh Budania
Re: upgrade from 3.4.5 to 3.5.6
Hi, Yes, My client and server both are using certificate and have added in ZK and client as well. Thanks, - Kuldeep Singh Budania On Thu, Oct 8, 2020 at 5:56 PM Enrico Olivelli wrote: > Il giorno gio 8 ott 2020 alle ore 14:17 kuldeep singh < > kuldeep.sing...@gmail.com> ha scritto: > > > Hi Team, > > > > I am facing one issue in SSL communication between client and zookeeper > > server. > > > > ZK 3.5.6 version > > > > 1. Mi on 3 node > > 2. Applying SSL and 3 nodes cluster is working fine > > 3. Scaled my cluster with 2 nodes and now my cluster have 5 nodes over > SSL > > > > but after scaling my SSL is not working between client and ZK server and > > even not able to login using zkCli as well. > > > > Can someone provide the details please why it is happening? > > > > Is your client configured to use SSL ? > > Enrico > > > > > > > Thanks, > > - > > Kuldeep Singh Budania > > Software Architect > > > > > > > > On Mon, Jul 13, 2020 at 2:19 PM Enrico Olivelli - Diennea > > wrote: > > > > > It looks like we ported it to 3.5. > > > > > > See the subtask > > > https://issues.apache.org/jira/browse/ZOOKEEPER-2792 > > > > > > Enrico > > > > > > Il giorno 13/07/20, 10:37 "kuldeep singh" > > ha > > > scritto: > > > > > > Hi Team, > > > > > > I appreciate it if I will get a response as soon as possible, as I > am > > > stuck > > > at this point. > > > > > > Thanks, > > > - > > > Kuldeep Singh Budania > > > > > > > > > > > > On Mon, Jul 13, 2020 at 11:10 AM kuldeep singh < > > > kuldeep.sing...@gmail.com> > > > wrote: > > > > > > > Hi Team, > > > > > > > > Server to Server communication is not supported in 3.5.6 version > > as > > > per > > > > below JIRA issue? > > > > > > > > https://issues.apache.org/jira/browse/ZOOKEEPER-2639 > > > > > > > > Thanks, > > > > - > > > > Kuldeep Singh Budania > > > > > > > > > > > > > > > > On Thu, Jul 2, 2020 at 4:24 PM kuldeep singh < > > > kuldeep.sing...@gmail.com> > > > > wrote: > > > > > > > >> Thanks for the reply. > > > >> > > > >> Now my ZKCli cmd is working fine as we use some our customized > > > >> authentication and we resolve the issue. > > > >> > > > >> Now I am going to implement Server to Server communication. > > > >> > > > >> Thanks, > > > >> - > > > >> Kuldeep Singh Budania > > > >> > > > >> > > > >> > > > >> On Thu, Jul 2, 2020 at 3:53 PM Szalay-Bekő Máté < > > > >> szalay.beko.m...@gmail.com> wrote: > > > >> > > > >>> I think SSL is working for you already... If you managed to > start > > > the > > > >>> zkCli.sh and be able to connect to ZooKeeper on the secure port > > > and issue > > > >>> any kind of command (like: " ls / "), then the wire encryption > is > > > working > > > >>> and your server/client communication is secured by ZooKeeper. > > > >>> > > > >>> Why you want to run the following command? > > > >>> addauth ztpasswd zooadmin: > > > >>> > > > >>> Do you also want to configure a superDigest user in ZooKeeper? > > > Please > > > >>> note > > > >>> that this command is independent from SSL. If you need to > create > > a > > > >>> username-password pair for digest authentication then please > use > > > the > > > >>> command in the following way: > > > >>> addauth digest zooadmin:yourSuperSecretPassword > > > >>> > > > >>> Kind regards, > > > >>> Mate > > > >>&g
Re: upgrade from 3.4.5 to 3.5.6
Hi Team, I am facing one issue in SSL communication between client and zookeeper server. ZK 3.5.6 version 1. Mi on 3 node 2. Applying SSL and 3 nodes cluster is working fine 3. Scaled my cluster with 2 nodes and now my cluster have 5 nodes over SSL but after scaling my SSL is not working between client and ZK server and even not able to login using zkCli as well. Can someone provide the details please why it is happening? Thanks, - Kuldeep Singh Budania Software Architect On Mon, Jul 13, 2020 at 2:19 PM Enrico Olivelli - Diennea wrote: > It looks like we ported it to 3.5. > > See the subtask > https://issues.apache.org/jira/browse/ZOOKEEPER-2792 > > Enrico > > Il giorno 13/07/20, 10:37 "kuldeep singh" ha > scritto: > > Hi Team, > > I appreciate it if I will get a response as soon as possible, as I am > stuck > at this point. > > Thanks, > - > Kuldeep Singh Budania > > > > On Mon, Jul 13, 2020 at 11:10 AM kuldeep singh < > kuldeep.sing...@gmail.com> > wrote: > > > Hi Team, > > > > Server to Server communication is not supported in 3.5.6 version as > per > > below JIRA issue? > > > > https://issues.apache.org/jira/browse/ZOOKEEPER-2639 > > > > Thanks, > > - > > Kuldeep Singh Budania > > > > > > > > On Thu, Jul 2, 2020 at 4:24 PM kuldeep singh < > kuldeep.sing...@gmail.com> > > wrote: > > > >> Thanks for the reply. > >> > >> Now my ZKCli cmd is working fine as we use some our customized > >> authentication and we resolve the issue. > >> > >> Now I am going to implement Server to Server communication. > >> > >> Thanks, > >> - > >> Kuldeep Singh Budania > >> > >> > >> > >> On Thu, Jul 2, 2020 at 3:53 PM Szalay-Bekő Máté < > >> szalay.beko.m...@gmail.com> wrote: > >> > >>> I think SSL is working for you already... If you managed to start > the > >>> zkCli.sh and be able to connect to ZooKeeper on the secure port > and issue > >>> any kind of command (like: " ls / "), then the wire encryption is > working > >>> and your server/client communication is secured by ZooKeeper. > >>> > >>> Why you want to run the following command? > >>> addauth ztpasswd zooadmin: > >>> > >>> Do you also want to configure a superDigest user in ZooKeeper? > Please > >>> note > >>> that this command is independent from SSL. If you need to create a > >>> username-password pair for digest authentication then please use > the > >>> command in the following way: > >>> addauth digest zooadmin:yourSuperSecretPassword > >>> > >>> Kind regards, > >>> Mate > >>> > >>> On Thu, Jul 2, 2020 at 6:59 AM kuldeep singh < > kuldeep.sing...@gmail.com> > >>> wrote: > >>> > >>> > 1. sh zkCli.sh --config /etc/zookeeper -server localhost:2281 > >>> > > >>> > 2. addauth ztpasswd zooadmin: > >>> > > >>> > > >>> > Thanks, > >>> > - > >>> > Kuldeep Singh Budania > >>> > > >>> > > >>> > > >>> > On Thu, Jul 2, 2020 at 9:56 AM kuldeep singh < > >>> kuldeep.sing...@gmail.com> > >>> > wrote: > >>> > > >>> > > Hi Team, > >>> > > > >>> > > Any update on this? > >>> > > > >>> > > Thanks, > >>> > > - > >>> > > Kuldeep Singh Budania > >>> > > > >>> > > > >>> > > > >>> > > On Wed, Jul 1, 2020 at 6:43 PM kuldeep singh < > >>> kuldeep.sing...@gmail.com> > >>> > > wrote: > >>> > > > >>> > >> Sorry this is my bad, there were server setting like below > >>> > >> > >>> > >> export SERVER_JVMFLAGS=" > >>
Re: upgrade from 3.4.5 to 3.5.6
Hi, more logs 2020-10-14 12:25:05,106 - ERROR [nioEventLoopGroup-7-4:NettyServerCnxnFactory$CnxnChannelHandler$CertificateVerifier@257] - Unsuccessful handshake with session 0x0 2020-10-14 12:25:05,107 - WARN [nioEventLoopGroup-7-4:NettyServerCnxnFactory$CnxnChannelHandler@138] - Exception caught io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 737276720a at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:475) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:283) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514) at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 737276720a at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1203) at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1271) at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:505) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:444) ... 17 more Thanks, - Kuldeep Singh Budania Software Architect On Wed, Oct 14, 2020 at 3:51 PM kuldeep singh wrote: > Hi, > > > below error is coming on zookeeper logs on SSL. > > 2020-10-14 12:18:27,410 - WARN [nioEventLoopGroup-4-16:NettyServerCnxn@540] > - Closing connection to /127.0.0.1:49470 > java.io.IOException: Len error 369296128 > at > org.apache.zookeeper.server.NettyServerCnxn.receiveMessage(NettyServerCnxn.java:533) > at > org.apache.zookeeper.server.NettyServerCnxn.processMessage(NettyServerCnxn.java:360) > at > org.apache.zookeeper.server.NettyServerCnxnFactory$CnxnChannelHandler.channelRead(NettyServerCnxnFactory.java:184) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) > at > io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) > at > io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931) > at > io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) > at > io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700) > at > io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635) > at > io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552) > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514) > at > io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044) > at > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) >
Re: upgrade from 3.4.5 to 3.5.6
Hi, Yes, all the configurations are the same on all 5 nodes. I have checked more than 10 times. Before scaling , 3 node cluster was working fine, but scaling from 3 to 5 node, then we are facing this problem. only question is on 3 node cluster it is working fine, but after scaling it is not working and zkcli is not able to connect. but one case I have seen. 1. if i remove all the SSL configuration from all the nodes and then applying again all SSL property then it is working fine. 2 cases 1st case :- 3 node cluster witout SSL is working fine scaled the cluster till 5 node without SSL is also working fine after that when applying SSL on all 5 nodes then it is also working 2nd case :- 3 node cluster with SSL is working fine scaled the cluster till 5 node with SSL is not working in both cases the configuration area same and certificate is also same. Thanks, - Kuldeep Singh Budania On Wed, Oct 14, 2020 at 5:41 PM Szalay-Bekő Máté wrote: > the config looks OK in general... > > - are you sure the same configs are used on all ZK servers? > - does the truststores accept all keys on the keystores? (if the > truststores of the old servers had to be modified, then did you restart the > old servers with the updated truststores?) > - did the 3 node ZK cluster work with SSL? (were you able to connect to it > with the client using SSL?) > > also: do you really need client authentication with SSL? (I see you are > using SASL too) > If you only need SSL for wire encryption, then you can try > with ssl.clientAuth=none (see the admin guide). Although that feature was > broken on 3.5.6, got fixed on 3.5.7 according to the doc. > > best regards, > Mate > > On Wed, Oct 14, 2020 at 1:10 PM kuldeep singh > wrote: > > > Sorry, > > secureClientPort=2182 > > > > Thanks, > > --------- > > Kuldeep Singh Budania > > > > > > > > On Wed, Oct 14, 2020 at 4:18 PM kuldeep singh > > > wrote: > > > > > Thanks for reply > > > > > > zoo.cfg > > > --- > > > secureClientPort=2181 > > > serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory > > > initLimit=10 > > > syncLimit=5 > > > dataDir= data directory (not mentioning exact path here) > > > tickTime=2000 > > > autopurge.snapRetainCount=3 > > > autopurge.purgeInterval=1 > > > admin.enableServer=false > > > standaloneEnabled=false > > > jute.maxbuffer=2147483648 > > > server.1=host1_priv:10288:10388 > > > server.2=host2_priv:10288:10388 > > > server.3=host3_priv:10288:10388 > > > server.4=host4_priv:10288:10388 > > > server.5=host5_priv:10288:10388 > > > quorum.auth.enableSasl=true > > > quorum.auth.learnerRequireSasl=true > > > quorum.auth.serverRequireSasl=true > > > quorum.auth.learner.loginContext=QuorumLearner > > > quorum.auth.server.loginContext=QuorumServer > > > quorum.cnxn.threads.size=10 > > > - > > > java.env > > > > > > export > > > > > > SERVER_JVMFLAGS="-Dzookeeper.serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory > > > -Dzookeeper.ssl.keyStore.location=keystore.jks > > > -Dzookeeper.ssl.keyStore.password= > > > -Dzookeeper.ssl.trustStore.location= keystore.jks > > > -Dzookeeper.ssl.trustStore.password= > > > -Djava.security.auth.login.config=zookeeper-jaas.conf" > > > > > > export > > > > > > CLIENT_JVMFLAGS="-Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty > > > -Dzookeeper.client.secure=true -Dzookeeper.ssl.keyStore.location= > > > keystore.jks -Dzookeeper.ssl.keyStore.password= > > > -Dzookeeper.ssl.trustStore.location=keystore.jks > > > -Dzookeeper.ssl.trustStore.password= > > > -Dzookeeper.ssl.hostnameVerification=false" > > > > > > Thanks, > > > - > > > Kuldeep Singh Budania > > > > > > > > > > > > On Wed, Oct 14, 2020 at 4:12 PM Szalay-Bekő Máté < > > > szalay.beko.m...@gmail.com> wrote: > > > > > >> These log messages indicate that a client (or an other ZooKeeper > server) > > >> is > > >> trying to connect without SSL to a ZooKeeper process that expects SSL. > > >> I assume this will be a configuration issue then. > > >> > > >&
Re: upgrade from 3.4.5 to 3.5.6
Hi, below error is coming on zookeeper logs on SSL. 2020-10-14 12:18:27,410 - WARN [nioEventLoopGroup-4-16:NettyServerCnxn@540] - Closing connection to /127.0.0.1:49470 java.io.IOException: Len error 369296128 at org.apache.zookeeper.server.NettyServerCnxn.receiveMessage(NettyServerCnxn.java:533) at org.apache.zookeeper.server.NettyServerCnxn.processMessage(NettyServerCnxn.java:360) at org.apache.zookeeper.server.NettyServerCnxnFactory$CnxnChannelHandler.channelRead(NettyServerCnxnFactory.java:184) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514) at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) Thanks, - Kuldeep Singh Budania On Thu, Oct 8, 2020 at 6:33 PM Szalay-Bekő Máté wrote: > Sounds like a bug or a configuration issue... > can you share the configs (before and after the scale-up) and the logs? > also: does the truststores recognise all the keys used on all the 5 > nodes? (e.g. the truststores on the old nodes accept the new keys?) > > Best Regards, > Mate > > On Thu, Oct 8, 2020 at 2:31 PM kuldeep singh > wrote: > > > > Hi, > > > > Yes, My client and server both are using certificate and have added in ZK > > and client as well. > > > > Thanks, > > - > > Kuldeep Singh Budania > > > > > > > > On Thu, Oct 8, 2020 at 5:56 PM Enrico Olivelli > wrote: > > > > > Il giorno gio 8 ott 2020 alle ore 14:17 kuldeep singh < > > > kuldeep.sing...@gmail.com> ha scritto: > > > > > > > Hi Team, > > > > > > > > I am facing one issue in SSL communication between client and > zookeeper > > > > server. > > > > > > > > ZK 3.5.6 version > > > > > > > > 1. Mi on 3 node > > > > 2. Applying SSL and 3 nodes cluster is working fine > > > > 3. Scaled my cluster with 2 nodes and now my cluster have 5 nodes > over > > > SSL > > > > > > > > but after scaling my SSL is not working between client and ZK server > and > > > > even not able to login using zkCli as well. > > > > > > > > Can someone provide the details please why it is happening? > > > > > > > > > > Is your client configured to use SSL ? > > > > > > Enrico > > > > > > > > > > > > > > > > > Thanks, > > > > - > > > > Kuldeep Singh Budania > > > > Software Architect > > > > > > > > > > > > > > > > On Mon, Jul 13, 2020 at 2:19 PM Enrico Olivelli - Diennea > > > > wrote: > > > > > > > > > It looks like we ported it to 3.5. > > > > > > > > > > See the subtask > > > > > https://issues.apache.org/jira/browse/ZOOKEEPER-2792 > > > > > > > > > > Enrico > > > > > > > > > > Il giorno 13/07/20, 10:37 "kuldeep singh" < > kuldeep.sing...@gmail.com> > > > > ha > > > > > scritto: > > > > > > > > > > Hi Team, > > > > > > > > >
Re: upgrade from 3.4.5 to 3.5.6
Sorry, secureClientPort=2182 Thanks, - Kuldeep Singh Budania On Wed, Oct 14, 2020 at 4:18 PM kuldeep singh wrote: > Thanks for reply > > zoo.cfg > --- > secureClientPort=2181 > serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory > initLimit=10 > syncLimit=5 > dataDir= data directory (not mentioning exact path here) > tickTime=2000 > autopurge.snapRetainCount=3 > autopurge.purgeInterval=1 > admin.enableServer=false > standaloneEnabled=false > jute.maxbuffer=2147483648 > server.1=host1_priv:10288:10388 > server.2=host2_priv:10288:10388 > server.3=host3_priv:10288:10388 > server.4=host4_priv:10288:10388 > server.5=host5_priv:10288:10388 > quorum.auth.enableSasl=true > quorum.auth.learnerRequireSasl=true > quorum.auth.serverRequireSasl=true > quorum.auth.learner.loginContext=QuorumLearner > quorum.auth.server.loginContext=QuorumServer > quorum.cnxn.threads.size=10 > - > java.env > > export > SERVER_JVMFLAGS="-Dzookeeper.serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory > -Dzookeeper.ssl.keyStore.location=keystore.jks > -Dzookeeper.ssl.keyStore.password= > -Dzookeeper.ssl.trustStore.location= keystore.jks > -Dzookeeper.ssl.trustStore.password= > -Djava.security.auth.login.config=zookeeper-jaas.conf" > > export > CLIENT_JVMFLAGS="-Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty > -Dzookeeper.client.secure=true -Dzookeeper.ssl.keyStore.location= > keystore.jks -Dzookeeper.ssl.keyStore.password= > -Dzookeeper.ssl.trustStore.location=keystore.jks > -Dzookeeper.ssl.trustStore.password= > -Dzookeeper.ssl.hostnameVerification=false" > > Thanks, > - > Kuldeep Singh Budania > > > > On Wed, Oct 14, 2020 at 4:12 PM Szalay-Bekő Máté < > szalay.beko.m...@gmail.com> wrote: > >> These log messages indicate that a client (or an other ZooKeeper server) >> is >> trying to connect without SSL to a ZooKeeper process that expects SSL. >> I assume this will be a configuration issue then. >> >> Best regards, >> Mate >> >> On Wed, Oct 14, 2020 at 12:30 PM kuldeep singh > > >> wrote: >> >> > Hi, >> > >> > more logs >> > >> > 2020-10-14 12:25:05,106 - ERROR >> > >> > >> [nioEventLoopGroup-7-4:NettyServerCnxnFactory$CnxnChannelHandler$CertificateVerifier@257 >> > ] >> > - Unsuccessful handshake with session 0x0 >> > >> > 2020-10-14 12:25:05,107 - WARN >> > [nioEventLoopGroup-7-4:NettyServerCnxnFactory$CnxnChannelHandler@138] - >> > Exception caught >> > >> > io.netty.handler.codec.DecoderException: >> > io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: >> > 737276720a >> > >> > at >> > >> > >> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:475) >> > >> > at >> > >> > >> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:283) >> > >> > at >> > >> > >> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) >> > >> > at >> > >> > >> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) >> > >> > at >> > >> > >> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) >> > >> > at >> > >> > >> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422) >> > >> > at >> > >> > >> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) >> > >> > at >> > >> > >> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) >> > >> > at >> > >> > >> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931) >> > >> > at >> > >> > >> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) >> > >> > at >> > >> io.netty.channel.nio.NioEventL
Re: upgrade from 3.4.5 to 3.5.6
Thanks for reply zoo.cfg --- secureClientPort=2181 serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory initLimit=10 syncLimit=5 dataDir= data directory (not mentioning exact path here) tickTime=2000 autopurge.snapRetainCount=3 autopurge.purgeInterval=1 admin.enableServer=false standaloneEnabled=false jute.maxbuffer=2147483648 server.1=host1_priv:10288:10388 server.2=host2_priv:10288:10388 server.3=host3_priv:10288:10388 server.4=host4_priv:10288:10388 server.5=host5_priv:10288:10388 quorum.auth.enableSasl=true quorum.auth.learnerRequireSasl=true quorum.auth.serverRequireSasl=true quorum.auth.learner.loginContext=QuorumLearner quorum.auth.server.loginContext=QuorumServer quorum.cnxn.threads.size=10 - java.env export SERVER_JVMFLAGS="-Dzookeeper.serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory -Dzookeeper.ssl.keyStore.location=keystore.jks -Dzookeeper.ssl.keyStore.password= -Dzookeeper.ssl.trustStore.location= keystore.jks -Dzookeeper.ssl.trustStore.password= -Djava.security.auth.login.config=zookeeper-jaas.conf" export CLIENT_JVMFLAGS="-Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty -Dzookeeper.client.secure=true -Dzookeeper.ssl.keyStore.location= keystore.jks -Dzookeeper.ssl.keyStore.password= -Dzookeeper.ssl.trustStore.location=keystore.jks -Dzookeeper.ssl.trustStore.password= -Dzookeeper.ssl.hostnameVerification=false" Thanks, --------- Kuldeep Singh Budania On Wed, Oct 14, 2020 at 4:12 PM Szalay-Bekő Máté wrote: > These log messages indicate that a client (or an other ZooKeeper server) is > trying to connect without SSL to a ZooKeeper process that expects SSL. > I assume this will be a configuration issue then. > > Best regards, > Mate > > On Wed, Oct 14, 2020 at 12:30 PM kuldeep singh > wrote: > > > Hi, > > > > more logs > > > > 2020-10-14 12:25:05,106 - ERROR > > > > > [nioEventLoopGroup-7-4:NettyServerCnxnFactory$CnxnChannelHandler$CertificateVerifier@257 > > ] > > - Unsuccessful handshake with session 0x0 > > > > 2020-10-14 12:25:05,107 - WARN > > [nioEventLoopGroup-7-4:NettyServerCnxnFactory$CnxnChannelHandler@138] - > > Exception caught > > > > io.netty.handler.codec.DecoderException: > > io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: > > 737276720a > > > > at > > > > > io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:475) > > > > at > > > > > io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:283) > > > > at > > > > > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) > > > > at > > > > > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) > > > > at > > > > > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) > > > > at > > > > > io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422) > > > > at > > > > > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) > > > > at > > > > > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) > > > > at > > > > > io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931) > > > > at > > > > > io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) > > > > at > > > io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700) > > > > at > > > > > io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635) > > > > at > > > > > io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552) > > > > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514) > > > > at > > > > > io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044) > > > > at > > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > > > > at > > > > > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > > > >
Re: upgrade from 3.4.5 to 3.5.6
Sorry this is my bad, there were server setting like below export SERVER_JVMFLAGS=" -Dzookeeper.serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory -Dzookeeper.ssl.keyStore.location=/root/zookeeper/ssl/testKeyStore.jks -Dzookeeper.ssl.keyStore.password=testpass -Dzookeeper.ssl.trustStore.location=/root/zookeeper/ssl/testTrustStore.jks -Dzookeeper.ssl.trustStore.password=testpass" export CLIENT_JVMFLAGS=" -Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty -Dzookeeper.client.secure=true -Dzookeeper.ssl.keyStore.location=/root/zookeeper/ssl/testKeyStore.jks -Dzookeeper.ssl.keyStore.password=testpass -Dzookeeper.ssl.trustStore.location=/root/zookeeper/ssl/testTrustStore.jks -Dzookeeper.ssl.trustStore.password=testpass" I want to have SSL between client to server communication I am already following the same link which you have shared with me but that is not working. Zoo.cfg secureClientPort=2281 initLimit=10 syncLimit=5 dataDir=/var/lib/zookeeper/data tickTime=2000 autopurge.snapRetainCount=3 autopurge.purgeInterval=1 admin.enableServer=false standaloneEnabled=false jute.maxbuffer=2147483648 serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory server.1=host1_priv:10288:10388 server.2=host2_priv:10288:10388 server.3=host3_priv:10288:10388 command to connect using zkcli 1. zkcli zoo.cfg localhost:2281 2. addauth ztpasswd usernaem:password after second step we are getting below error WatchedEvent state:AuthFailed type:None path:null Zookeeper logs :- *2020-07-01 07:38:09,342 - WARN [nioEventLoopGroup-4-2:ZooKeeperServer@1119] - No authentication provider for scheme: ztpasswd has x509 ip digest* Thanks, --------- Kuldeep Singh Budania On Wed, Jul 1, 2020 at 6:25 PM Szalay-Bekő Máté wrote: > > No authentication provider for scheme: ztpasswd has x509 ip digest* > > This suggest you have some configuration error... Where did you use the > "ztpasswd" string in your configs / commands? > > On Wed, Jul 1, 2020 at 2:53 PM Szalay-Bekő Máté < > szalay.beko.m...@gmail.com> > wrote: > > > > My ZK server is up and running in secure mode > > > > What is your goal? You want to setup client-server SSL connection? > > > > see: > > > https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZooKeeper+SSL+User+Guide > > > > (or you want to have both SSL and SASL enabled?) > > > > Anyway, please remove the following line from the SERVER_JVMFLAGS: > > -Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty > > This is a configuration that makes sense only for the ZooKeeper client, > > not for the server. For the server, use the following: > > > > > -Dzookeeper.serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory > > > > If it doesn't solve the issue, then can you please send your zoo.cfg > file? > > Also can you please send the zkCli command you execute? (you need to > > connect to the secure ZooKeeper port, unless portUnification is enabled) > > > > Kind regards, > > Mate > > > > On Wed, Jul 1, 2020 at 9:48 AM kuldeep singh > > wrote: > > > >> Hi, > >> > >> we have done below changes in java.env file > >> > >> export SERVER_JVMFLAGS=" > >> > >> -Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty > >> -Dzookeeper.ssl.keyStore.location=/root/zookeeper/ssl/testKeyStore.jks > >> -Dzookeeper.ssl.keyStore.password=testpass > >> > -Dzookeeper.ssl.trustStore.location=/root/zookeeper/ssl/testTrustStore.jks > >> -Dzookeeper.ssl.trustStore.password=testpass" > >> > >> > >> > >> export CLIENT_JVMFLAGS=" > >> > >> -Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty > >> -Dzookeeper.client.secure=true > >> -Dzookeeper.ssl.keyStore.location=/root/zookeeper/ssl/testKeyStore.jks > >> -Dzookeeper.ssl.keyStore.password=testpass > >> > -Dzookeeper.ssl.trustStore.location=/root/zookeeper/ssl/testTrustStore.jks > >> -Dzookeeper.ssl.trustStore.password=testpass" > >> > >> I have started the ZK server and it is up without any issue. > >> > >> But now when I login to ZkCli then it gives the below error. > >> > >> WatchedEvent state:AuthFailed type:None path:null > >> > >> > >> Zookeeper logs :- *2020-07-01 07:38:09,342 - WARN > >> [nioEventLoopGroup-4-2:ZooKeeperServer@1119] - No authentication > provider > >> for scheme: ztpasswd has x509 ip digest* > >> > >> Please help me on this issue > >> > >> Thanks,
Re: upgrade from 3.4.5 to 3.5.6
Hi Team, I appreciate it if I will get a response as soon as possible, as I am stuck at this point. Thanks, - Kuldeep Singh Budania On Mon, Jul 13, 2020 at 11:10 AM kuldeep singh wrote: > Hi Team, > > Server to Server communication is not supported in 3.5.6 version as per > below JIRA issue? > > https://issues.apache.org/jira/browse/ZOOKEEPER-2639 > > Thanks, > ----- > Kuldeep Singh Budania > > > > On Thu, Jul 2, 2020 at 4:24 PM kuldeep singh > wrote: > >> Thanks for the reply. >> >> Now my ZKCli cmd is working fine as we use some our customized >> authentication and we resolve the issue. >> >> Now I am going to implement Server to Server communication. >> >> Thanks, >> - >> Kuldeep Singh Budania >> >> >> >> On Thu, Jul 2, 2020 at 3:53 PM Szalay-Bekő Máté < >> szalay.beko.m...@gmail.com> wrote: >> >>> I think SSL is working for you already... If you managed to start the >>> zkCli.sh and be able to connect to ZooKeeper on the secure port and issue >>> any kind of command (like: " ls / "), then the wire encryption is working >>> and your server/client communication is secured by ZooKeeper. >>> >>> Why you want to run the following command? >>> addauth ztpasswd zooadmin: >>> >>> Do you also want to configure a superDigest user in ZooKeeper? Please >>> note >>> that this command is independent from SSL. If you need to create a >>> username-password pair for digest authentication then please use the >>> command in the following way: >>> addauth digest zooadmin:yourSuperSecretPassword >>> >>> Kind regards, >>> Mate >>> >>> On Thu, Jul 2, 2020 at 6:59 AM kuldeep singh >>> wrote: >>> >>> > 1. sh zkCli.sh --config /etc/zookeeper -server localhost:2281 >>> > >>> > 2. addauth ztpasswd zooadmin: >>> > >>> > >>> > Thanks, >>> > - >>> > Kuldeep Singh Budania >>> > >>> > >>> > >>> > On Thu, Jul 2, 2020 at 9:56 AM kuldeep singh < >>> kuldeep.sing...@gmail.com> >>> > wrote: >>> > >>> > > Hi Team, >>> > > >>> > > Any update on this? >>> > > >>> > > Thanks, >>> > > - >>> > > Kuldeep Singh Budania >>> > > >>> > > >>> > > >>> > > On Wed, Jul 1, 2020 at 6:43 PM kuldeep singh < >>> kuldeep.sing...@gmail.com> >>> > > wrote: >>> > > >>> > >> Sorry this is my bad, there were server setting like below >>> > >> >>> > >> export SERVER_JVMFLAGS=" >>> > >> >>> > >> >>> > >> >>> > >>> -Dzookeeper.serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory >>> > >> >>> > >> >>> -Dzookeeper.ssl.keyStore.location=/root/zookeeper/ssl/testKeyStore.jks >>> > >> -Dzookeeper.ssl.keyStore.password=testpass >>> > >> >>> > >>> -Dzookeeper.ssl.trustStore.location=/root/zookeeper/ssl/testTrustStore.jks >>> > >> -Dzookeeper.ssl.trustStore.password=testpass" >>> > >> >>> > >> >>> > >> >>> > >> export CLIENT_JVMFLAGS=" >>> > >> >>> > >> >>> -Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty >>> > >> -Dzookeeper.client.secure=true >>> > >> >>> -Dzookeeper.ssl.keyStore.location=/root/zookeeper/ssl/testKeyStore.jks >>> > >> -Dzookeeper.ssl.keyStore.password=testpass >>> > >> >>> > >>> -Dzookeeper.ssl.trustStore.location=/root/zookeeper/ssl/testTrustStore.jks >>> > >> -Dzookeeper.ssl.trustStore.password=testpass" >>> > >> >>> > >> I want to have SSL between client to server communication >>> > >> >>> > >> I am already following the same link which you have shared with me >>> but >>> > >> that is not working. >>> > >> >>> > >> Zoo.cfg >>> > >> >>> > >> secureClientPort=2281 >>> > >> initLimit=10 >>> &g
Re: upgrade from 3.4.5 to 3.5.6
Thanks for the reply. Now my ZKCli cmd is working fine as we use some our customized authentication and we resolve the issue. Now I am going to implement Server to Server communication. Thanks, - Kuldeep Singh Budania On Thu, Jul 2, 2020 at 3:53 PM Szalay-Bekő Máté wrote: > I think SSL is working for you already... If you managed to start the > zkCli.sh and be able to connect to ZooKeeper on the secure port and issue > any kind of command (like: " ls / "), then the wire encryption is working > and your server/client communication is secured by ZooKeeper. > > Why you want to run the following command? > addauth ztpasswd zooadmin: > > Do you also want to configure a superDigest user in ZooKeeper? Please note > that this command is independent from SSL. If you need to create a > username-password pair for digest authentication then please use the > command in the following way: > addauth digest zooadmin:yourSuperSecretPassword > > Kind regards, > Mate > > On Thu, Jul 2, 2020 at 6:59 AM kuldeep singh > wrote: > > > 1. sh zkCli.sh --config /etc/zookeeper -server localhost:2281 > > > > 2. addauth ztpasswd zooadmin: > > > > > > Thanks, > > - > > Kuldeep Singh Budania > > > > > > > > On Thu, Jul 2, 2020 at 9:56 AM kuldeep singh > > wrote: > > > > > Hi Team, > > > > > > Any update on this? > > > > > > Thanks, > > > - > > > Kuldeep Singh Budania > > > > > > > > > > > > On Wed, Jul 1, 2020 at 6:43 PM kuldeep singh < > kuldeep.sing...@gmail.com> > > > wrote: > > > > > >> Sorry this is my bad, there were server setting like below > > >> > > >> export SERVER_JVMFLAGS=" > > >> > > >> > > >> > > > -Dzookeeper.serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory > > >> > > >> -Dzookeeper.ssl.keyStore.location=/root/zookeeper/ssl/testKeyStore.jks > > >> -Dzookeeper.ssl.keyStore.password=testpass > > >> > > > -Dzookeeper.ssl.trustStore.location=/root/zookeeper/ssl/testTrustStore.jks > > >> -Dzookeeper.ssl.trustStore.password=testpass" > > >> > > >> > > >> > > >> export CLIENT_JVMFLAGS=" > > >> > > >> > -Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty > > >> -Dzookeeper.client.secure=true > > >> -Dzookeeper.ssl.keyStore.location=/root/zookeeper/ssl/testKeyStore.jks > > >> -Dzookeeper.ssl.keyStore.password=testpass > > >> > > > -Dzookeeper.ssl.trustStore.location=/root/zookeeper/ssl/testTrustStore.jks > > >> -Dzookeeper.ssl.trustStore.password=testpass" > > >> > > >> I want to have SSL between client to server communication > > >> > > >> I am already following the same link which you have shared with me but > > >> that is not working. > > >> > > >> Zoo.cfg > > >> > > >> secureClientPort=2281 > > >> initLimit=10 > > >> syncLimit=5 > > >> dataDir=/var/lib/zookeeper/data > > >> tickTime=2000 > > >> autopurge.snapRetainCount=3 > > >> autopurge.purgeInterval=1 > > >> admin.enableServer=false > > >> standaloneEnabled=false > > >> jute.maxbuffer=2147483648 > > >> serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory > > >> server.1=host1_priv:10288:10388 > > >> server.2=host2_priv:10288:10388 > > >> server.3=host3_priv:10288:10388 > > >> > > >> > > >> command to connect using zkcli > > >> > > >> 1. zkcli zoo.cfg localhost:2281 > > >> 2. addauth ztpasswd usernaem:password > > >> > > >> after second step we are getting below error > > >> > > >> WatchedEvent state:AuthFailed type:None path:null > > >> > > >> > > >> Zookeeper logs :- *2020-07-01 07:38:09,342 - WARN > > >> [nioEventLoopGroup-4-2:ZooKeeperServer@1119] - No authentication > > provider > > >> for scheme: ztpasswd has x509 ip digest* > > >> > > >> Thanks, > > >> - > > >> Kuldeep Singh Budania > > >> > > >> > > >> > > >> On Wed, Jul 1, 2020 at 6:25 PM Szalay-Bekő Máté < > > >> szalay.beko.m...@g
Re: upgrade from 3.4.5 to 3.5.6
Hi Team, Server to Server communication is not supported in 3.5.6 version as per below JIRA issue? https://issues.apache.org/jira/browse/ZOOKEEPER-2639 Thanks, - Kuldeep Singh Budania On Thu, Jul 2, 2020 at 4:24 PM kuldeep singh wrote: > Thanks for the reply. > > Now my ZKCli cmd is working fine as we use some our customized > authentication and we resolve the issue. > > Now I am going to implement Server to Server communication. > > Thanks, > ----- > Kuldeep Singh Budania > > > > On Thu, Jul 2, 2020 at 3:53 PM Szalay-Bekő Máté < > szalay.beko.m...@gmail.com> wrote: > >> I think SSL is working for you already... If you managed to start the >> zkCli.sh and be able to connect to ZooKeeper on the secure port and issue >> any kind of command (like: " ls / "), then the wire encryption is working >> and your server/client communication is secured by ZooKeeper. >> >> Why you want to run the following command? >> addauth ztpasswd zooadmin: >> >> Do you also want to configure a superDigest user in ZooKeeper? Please note >> that this command is independent from SSL. If you need to create a >> username-password pair for digest authentication then please use the >> command in the following way: >> addauth digest zooadmin:yourSuperSecretPassword >> >> Kind regards, >> Mate >> >> On Thu, Jul 2, 2020 at 6:59 AM kuldeep singh >> wrote: >> >> > 1. sh zkCli.sh --config /etc/zookeeper -server localhost:2281 >> > >> > 2. addauth ztpasswd zooadmin: >> > >> > >> > Thanks, >> > ----- >> > Kuldeep Singh Budania >> > >> > >> > >> > On Thu, Jul 2, 2020 at 9:56 AM kuldeep singh > > >> > wrote: >> > >> > > Hi Team, >> > > >> > > Any update on this? >> > > >> > > Thanks, >> > > - >> > > Kuldeep Singh Budania >> > > >> > > >> > > >> > > On Wed, Jul 1, 2020 at 6:43 PM kuldeep singh < >> kuldeep.sing...@gmail.com> >> > > wrote: >> > > >> > >> Sorry this is my bad, there were server setting like below >> > >> >> > >> export SERVER_JVMFLAGS=" >> > >> >> > >> >> > >> >> > >> -Dzookeeper.serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory >> > >> >> > >> >> -Dzookeeper.ssl.keyStore.location=/root/zookeeper/ssl/testKeyStore.jks >> > >> -Dzookeeper.ssl.keyStore.password=testpass >> > >> >> > >> -Dzookeeper.ssl.trustStore.location=/root/zookeeper/ssl/testTrustStore.jks >> > >> -Dzookeeper.ssl.trustStore.password=testpass" >> > >> >> > >> >> > >> >> > >> export CLIENT_JVMFLAGS=" >> > >> >> > >> >> -Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty >> > >> -Dzookeeper.client.secure=true >> > >> >> -Dzookeeper.ssl.keyStore.location=/root/zookeeper/ssl/testKeyStore.jks >> > >> -Dzookeeper.ssl.keyStore.password=testpass >> > >> >> > >> -Dzookeeper.ssl.trustStore.location=/root/zookeeper/ssl/testTrustStore.jks >> > >> -Dzookeeper.ssl.trustStore.password=testpass" >> > >> >> > >> I want to have SSL between client to server communication >> > >> >> > >> I am already following the same link which you have shared with me >> but >> > >> that is not working. >> > >> >> > >> Zoo.cfg >> > >> >> > >> secureClientPort=2281 >> > >> initLimit=10 >> > >> syncLimit=5 >> > >> dataDir=/var/lib/zookeeper/data >> > >> tickTime=2000 >> > >> autopurge.snapRetainCount=3 >> > >> autopurge.purgeInterval=1 >> > >> admin.enableServer=false >> > >> standaloneEnabled=false >> > >> jute.maxbuffer=2147483648 >> > >> serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory >> > >> server.1=host1_priv:10288:10388 >> > >> server.2=host2_priv:10288:10388 >> > >> server.3=host3_priv:10288:10388 >> > >> >> > >> >> > >> command to connect using zkcli >> > >> >> > >> 1.
Re: upgrade from 3.4.5 to 3.5.6
Hi, My ZK server is up and running in secure mode, But When I am trying to connect to the ZK server using ZKCli, it gives the below error. WatchedEvent state:AuthFailed type:None path:null Zookeeper logs :- *2020-07-01 07:38:09,342 - WARN [nioEventLoopGroup-4-2:ZooKeeperServer@1119] - No authentication provider for scheme: ztpasswd has x509 ip digest* Can someone please help me on this issue. we are using the 3.5.6 version. I appreciate if I will get a response as soon as possible, as I am stuck at this point. Thanks, - Kuldeep Singh Budania Software Architect On Thu, Jun 25, 2020 at 11:54 AM Enrico Olivelli - Diennea wrote: > I mean in zoo.cfg > Not as a system property > > Enrico > > Il giorno 25/06/20, 08:19 "Enrico Olivelli - Diennea" < > enrico.olive...@diennea.com.INVALID> ha scritto: > > Hi > You have to enable Netty on the server side > > Something like: > serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory > > Hope that helps > Enrico > > Il giorno 24/06/20, 19:17 "kuldeep singh" > ha scritto: > > Hi, > > I got below error while setting SSL properties in zkEnv.sh > > > > > == > > 2020-06-24 15:49:35,864 - INFO [main:QuorumPeerConfig@133] - > Reading > configuration from: /etc/zookeeper/zoo.cfg > > 2020-06-24 15:49:35,874 - INFO [main:QuorumPeerConfig@385] - > clientPortAddress is 0.0.0.0/0.0.0.0:10181 > > 2020-06-24 15:49:35,874 - INFO [main:QuorumPeerConfig@399] - > secureClientPortAddress is 0.0.0.0/0.0.0.0:2281 > > 2020-06-24 15:49:35,878 - INFO [main:X509Util@79] - Setting -D > jdk.tls.rejectClientInitiatedRenegotiation=true to disable > client-initiated > TLS renegotiation > > 2020-06-24 15:49:35,897 - INFO [main:DatadirCleanupManager@78] - > autopurge.snapRetainCount set to 3 > > 2020-06-24 15:49:35,897 - INFO [main:DatadirCleanupManager@79] - > autopurge.purgeInterval set to 1 > > 2020-06-24 15:49:35,898 - INFO [ > PurgeTask:DatadirCleanupManager$PurgeTask@138] - Purge task > started. > > 2020-06-24 15:49:35,899 - INFO [main:ManagedUtil@46] - Log4j > found with > jmx enabled. > > 2020-06-24 15:49:35,903 - INFO [PurgeTask:FileTxnSnapLog@103] - > zookeeper.snapshot.trust.empty : false > > 2020-06-24 15:49:35,910 - INFO [ > PurgeTask:DatadirCleanupManager$PurgeTask@144] - Purge task > completed. > > 2020-06-24 15:49:35,975 - INFO [main:QuorumPeerMain@141] - > Starting quorum > peer > > 2020-06-24 15:49:35,983 - INFO [main:ServerCnxnFactory@135] - > Using > org.apache.zookeeper.server.NIOServerCnxnFactory as server > connection > factory > > 2020-06-24 15:49:35,986 - INFO [main:NIOServerCnxnFactory@673] - > Configuring NIO connection handler with 10s sessionless connection > timeout, > 2 selector thread(s), 16 worker threads, and 64 kB direct buffers. > > 2020-06-24 15:49:35,992 - INFO [main:NIOServerCnxnFactory@686] - > binding > to port 0.0.0.0/0.0.0.0:10181 > > 2020-06-24 15:49:35,994 - INFO [main:ServerCnxnFactory@135] - > Using > org.apache.zookeeper.server.NIOServerCnxnFactory as server > connection > factory > > 2020-06-24 15:49:35,995 - ERROR [main:QuorumPeerMain@101] - > Unexpected > exception, exiting abnormally > > java.lang.UnsupportedOperationException: SSL isn't supported in > NIOServerCnxn > > at > > org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:644) > > at > > org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:155) > > at > > org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:123) > > at > > org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82) > > > > > > I have set the following properties in SERVER_JVMFLAGS in zkEnv.sh > file : > > "-Dzookeeper.serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory > > -Dzookeeper.ssl.keyStore.location=/var/opt/vs/SecureInterface/keystore/CassSpkkeystore.p12 > -Dzookeeper.ssl.keyStore.password=EvaiKiO1@123456 > > -Dzookeeper.ssl.trustStore.location=/var/opt/vs/SecureInterface/keystore/CassSpkTrustStore.jks
Re: upgrade from 3.4.5 to 3.5.6
Hi Team, Any update on this? Thanks, - Kuldeep Singh Budania On Wed, Jul 1, 2020 at 6:43 PM kuldeep singh wrote: > Sorry this is my bad, there were server setting like below > > export SERVER_JVMFLAGS=" > > > -Dzookeeper.serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory > > -Dzookeeper.ssl.keyStore.location=/root/zookeeper/ssl/testKeyStore.jks > -Dzookeeper.ssl.keyStore.password=testpass > -Dzookeeper.ssl.trustStore.location=/root/zookeeper/ssl/testTrustStore.jks > -Dzookeeper.ssl.trustStore.password=testpass" > > > > export CLIENT_JVMFLAGS=" > > -Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty > -Dzookeeper.client.secure=true > -Dzookeeper.ssl.keyStore.location=/root/zookeeper/ssl/testKeyStore.jks > -Dzookeeper.ssl.keyStore.password=testpass > -Dzookeeper.ssl.trustStore.location=/root/zookeeper/ssl/testTrustStore.jks > -Dzookeeper.ssl.trustStore.password=testpass" > > I want to have SSL between client to server communication > > I am already following the same link which you have shared with me but > that is not working. > > Zoo.cfg > > secureClientPort=2281 > initLimit=10 > syncLimit=5 > dataDir=/var/lib/zookeeper/data > tickTime=2000 > autopurge.snapRetainCount=3 > autopurge.purgeInterval=1 > admin.enableServer=false > standaloneEnabled=false > jute.maxbuffer=2147483648 > serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory > server.1=host1_priv:10288:10388 > server.2=host2_priv:10288:10388 > server.3=host3_priv:10288:10388 > > > command to connect using zkcli > > 1. zkcli zoo.cfg localhost:2281 > 2. addauth ztpasswd usernaem:password > > after second step we are getting below error > > WatchedEvent state:AuthFailed type:None path:null > > > Zookeeper logs :- *2020-07-01 07:38:09,342 - WARN > [nioEventLoopGroup-4-2:ZooKeeperServer@1119] - No authentication provider > for scheme: ztpasswd has x509 ip digest* > > Thanks, > - > Kuldeep Singh Budania > > > > On Wed, Jul 1, 2020 at 6:25 PM Szalay-Bekő Máté < > szalay.beko.m...@gmail.com> wrote: > >> > No authentication provider for scheme: ztpasswd has x509 ip digest* >> >> This suggest you have some configuration error... Where did you use the >> "ztpasswd" string in your configs / commands? >> >> On Wed, Jul 1, 2020 at 2:53 PM Szalay-Bekő Máté < >> szalay.beko.m...@gmail.com> >> wrote: >> >> > > My ZK server is up and running in secure mode >> > >> > What is your goal? You want to setup client-server SSL connection? >> > >> > see: >> > >> https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZooKeeper+SSL+User+Guide >> > >> > (or you want to have both SSL and SASL enabled?) >> > >> > Anyway, please remove the following line from the SERVER_JVMFLAGS: >> > -Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty >> > This is a configuration that makes sense only for the ZooKeeper client, >> > not for the server. For the server, use the following: >> > >> > >> -Dzookeeper.serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory >> > >> > If it doesn't solve the issue, then can you please send your zoo.cfg >> file? >> > Also can you please send the zkCli command you execute? (you need to >> > connect to the secure ZooKeeper port, unless portUnification is enabled) >> > >> > Kind regards, >> > Mate >> > >> > On Wed, Jul 1, 2020 at 9:48 AM kuldeep singh > > >> > wrote: >> > >> >> Hi, >> >> >> >> we have done below changes in java.env file >> >> >> >> export SERVER_JVMFLAGS=" >> >> >> >> -Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty >> >> -Dzookeeper.ssl.keyStore.location=/root/zookeeper/ssl/testKeyStore.jks >> >> -Dzookeeper.ssl.keyStore.password=testpass >> >> >> -Dzookeeper.ssl.trustStore.location=/root/zookeeper/ssl/testTrustStore.jks >> >> -Dzookeeper.ssl.trustStore.password=testpass" >> >> >> >> >> >> >> >> export CLIENT_JVMFLAGS=" >> >> >> >> -Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty >> >> -Dzookeeper.client.secure=true >> >> -Dzookeeper.ssl.keyStore.location=/root/zookeeper/ssl/testKeyStore.jks >> >> -Dzookeeper.ssl.keyStore.password=testpass >> &g
Re: upgrade from 3.4.5 to 3.5.6
1. sh zkCli.sh --config /etc/zookeeper -server localhost:2281 2. addauth ztpasswd zooadmin: Thanks, - Kuldeep Singh Budania On Thu, Jul 2, 2020 at 9:56 AM kuldeep singh wrote: > Hi Team, > > Any update on this? > > Thanks, > --------- > Kuldeep Singh Budania > > > > On Wed, Jul 1, 2020 at 6:43 PM kuldeep singh > wrote: > >> Sorry this is my bad, there were server setting like below >> >> export SERVER_JVMFLAGS=" >> >> >> -Dzookeeper.serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory >> >> -Dzookeeper.ssl.keyStore.location=/root/zookeeper/ssl/testKeyStore.jks >> -Dzookeeper.ssl.keyStore.password=testpass >> -Dzookeeper.ssl.trustStore.location=/root/zookeeper/ssl/testTrustStore.jks >> -Dzookeeper.ssl.trustStore.password=testpass" >> >> >> >> export CLIENT_JVMFLAGS=" >> >> -Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty >> -Dzookeeper.client.secure=true >> -Dzookeeper.ssl.keyStore.location=/root/zookeeper/ssl/testKeyStore.jks >> -Dzookeeper.ssl.keyStore.password=testpass >> -Dzookeeper.ssl.trustStore.location=/root/zookeeper/ssl/testTrustStore.jks >> -Dzookeeper.ssl.trustStore.password=testpass" >> >> I want to have SSL between client to server communication >> >> I am already following the same link which you have shared with me but >> that is not working. >> >> Zoo.cfg >> >> secureClientPort=2281 >> initLimit=10 >> syncLimit=5 >> dataDir=/var/lib/zookeeper/data >> tickTime=2000 >> autopurge.snapRetainCount=3 >> autopurge.purgeInterval=1 >> admin.enableServer=false >> standaloneEnabled=false >> jute.maxbuffer=2147483648 >> serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory >> server.1=host1_priv:10288:10388 >> server.2=host2_priv:10288:10388 >> server.3=host3_priv:10288:10388 >> >> >> command to connect using zkcli >> >> 1. zkcli zoo.cfg localhost:2281 >> 2. addauth ztpasswd usernaem:password >> >> after second step we are getting below error >> >> WatchedEvent state:AuthFailed type:None path:null >> >> >> Zookeeper logs :- *2020-07-01 07:38:09,342 - WARN >> [nioEventLoopGroup-4-2:ZooKeeperServer@1119] - No authentication provider >> for scheme: ztpasswd has x509 ip digest* >> >> Thanks, >> - >> Kuldeep Singh Budania >> >> >> >> On Wed, Jul 1, 2020 at 6:25 PM Szalay-Bekő Máté < >> szalay.beko.m...@gmail.com> wrote: >> >>> > No authentication provider for scheme: ztpasswd has x509 ip digest* >>> >>> This suggest you have some configuration error... Where did you use the >>> "ztpasswd" string in your configs / commands? >>> >>> On Wed, Jul 1, 2020 at 2:53 PM Szalay-Bekő Máté < >>> szalay.beko.m...@gmail.com> >>> wrote: >>> >>> > > My ZK server is up and running in secure mode >>> > >>> > What is your goal? You want to setup client-server SSL connection? >>> > >>> > see: >>> > >>> https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZooKeeper+SSL+User+Guide >>> > >>> > (or you want to have both SSL and SASL enabled?) >>> > >>> > Anyway, please remove the following line from the SERVER_JVMFLAGS: >>> > -Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty >>> > This is a configuration that makes sense only for the ZooKeeper client, >>> > not for the server. For the server, use the following: >>> > >>> > >>> -Dzookeeper.serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory >>> > >>> > If it doesn't solve the issue, then can you please send your zoo.cfg >>> file? >>> > Also can you please send the zkCli command you execute? (you need to >>> > connect to the secure ZooKeeper port, unless portUnification is >>> enabled) >>> > >>> > Kind regards, >>> > Mate >>> > >>> > On Wed, Jul 1, 2020 at 9:48 AM kuldeep singh < >>> kuldeep.sing...@gmail.com> >>> > wrote: >>> > >>> >> Hi, >>> >> >>> >> we have done below changes in java.env file >>> >> >>> >> export SERVER_JVMFLAGS=" >>> >> >>> >> >>> -Dzooke
Re: upgrade from 3.4.5 to 3.5.6
Hi, we have done below changes in java.env file export SERVER_JVMFLAGS=" -Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty -Dzookeeper.ssl.keyStore.location=/root/zookeeper/ssl/testKeyStore.jks -Dzookeeper.ssl.keyStore.password=testpass -Dzookeeper.ssl.trustStore.location=/root/zookeeper/ssl/testTrustStore.jks -Dzookeeper.ssl.trustStore.password=testpass" export CLIENT_JVMFLAGS=" -Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty -Dzookeeper.client.secure=true -Dzookeeper.ssl.keyStore.location=/root/zookeeper/ssl/testKeyStore.jks -Dzookeeper.ssl.keyStore.password=testpass -Dzookeeper.ssl.trustStore.location=/root/zookeeper/ssl/testTrustStore.jks -Dzookeeper.ssl.trustStore.password=testpass" I have started the ZK server and it is up without any issue. But now when I login to ZkCli then it gives the below error. WatchedEvent state:AuthFailed type:None path:null Zookeeper logs :- *2020-07-01 07:38:09,342 - WARN [nioEventLoopGroup-4-2:ZooKeeperServer@1119] - No authentication provider for scheme: ztpasswd has x509 ip digest* Please help me on this issue Thanks, --------- Kuldeep Singh Budania Software Architect On Wed, Jul 1, 2020 at 12:05 PM kuldeep singh wrote: > Hi, > > My ZK server is up and running in secure mode, But When I am trying to > connect to the ZK server using ZKCli, it gives the below error. > > WatchedEvent state:AuthFailed type:None path:null > > > Zookeeper logs :- *2020-07-01 07:38:09,342 - WARN > [nioEventLoopGroup-4-2:ZooKeeperServer@1119] - No authentication provider > for scheme: ztpasswd has x509 ip digest* > > Can someone please help me on this issue. we are using the 3.5.6 version. > > I appreciate if I will get a response as soon as possible, as I am stuck > at this point. > > Thanks, > - > Kuldeep Singh Budania > Software Architect > > > > On Thu, Jun 25, 2020 at 11:54 AM Enrico Olivelli - Diennea > wrote: > >> I mean in zoo.cfg >> Not as a system property >> >> Enrico >> >> Il giorno 25/06/20, 08:19 "Enrico Olivelli - Diennea" < >> enrico.olive...@diennea.com.INVALID> ha scritto: >> >> Hi >> You have to enable Netty on the server side >> >> Something like: >> serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory >> >> Hope that helps >> Enrico >> >> Il giorno 24/06/20, 19:17 "kuldeep singh" >> ha scritto: >> >> Hi, >> >> I got below error while setting SSL properties in zkEnv.sh >> >> >> >> >> == >> >> 2020-06-24 15:49:35,864 - INFO [main:QuorumPeerConfig@133] - >> Reading >> configuration from: /etc/zookeeper/zoo.cfg >> >> 2020-06-24 15:49:35,874 - INFO [main:QuorumPeerConfig@385] - >> clientPortAddress is 0.0.0.0/0.0.0.0:10181 >> >> 2020-06-24 15:49:35,874 - INFO [main:QuorumPeerConfig@399] - >> secureClientPortAddress is 0.0.0.0/0.0.0.0:2281 >> >> 2020-06-24 15:49:35,878 - INFO [main:X509Util@79] - Setting -D >> jdk.tls.rejectClientInitiatedRenegotiation=true to disable >> client-initiated >> TLS renegotiation >> >> 2020-06-24 15:49:35,897 - INFO [main:DatadirCleanupManager@78] - >> autopurge.snapRetainCount set to 3 >> >> 2020-06-24 15:49:35,897 - INFO [main:DatadirCleanupManager@79] - >> autopurge.purgeInterval set to 1 >> >> 2020-06-24 15:49:35,898 - INFO [ >> PurgeTask:DatadirCleanupManager$PurgeTask@138] - Purge task >> started. >> >> 2020-06-24 15:49:35,899 - INFO [main:ManagedUtil@46] - Log4j >> found with >> jmx enabled. >> >> 2020-06-24 15:49:35,903 - INFO [PurgeTask:FileTxnSnapLog@103] - >> zookeeper.snapshot.trust.empty : false >> >> 2020-06-24 15:49:35,910 - INFO [ >> PurgeTask:DatadirCleanupManager$PurgeTask@144] - Purge task >> completed. >> >> 2020-06-24 15:49:35,975 - INFO [main:QuorumPeerMain@141] - >> Starting quorum >> peer >> >> 2020-06-24 15:49:35,983 - INFO [main:ServerCnxnFactory@135] - >> Using >> org.apache.zookeeper.server.NIOServerCnxnFactory as server >> connection >> factory >> >> 2020-06-24 15:49:35,986 - INFO [main:NIOServerCnxnFactory@673] - >> Configuring NIO c
Re: upgrade from 3.4.5 to 3.5.6
Hi, I got below error while setting SSL properties in zkEnv.sh == 2020-06-24 15:49:35,864 - INFO [main:QuorumPeerConfig@133] - Reading configuration from: /etc/zookeeper/zoo.cfg 2020-06-24 15:49:35,874 - INFO [main:QuorumPeerConfig@385] - clientPortAddress is 0.0.0.0/0.0.0.0:10181 2020-06-24 15:49:35,874 - INFO [main:QuorumPeerConfig@399] - secureClientPortAddress is 0.0.0.0/0.0.0.0:2281 2020-06-24 15:49:35,878 - INFO [main:X509Util@79] - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation 2020-06-24 15:49:35,897 - INFO [main:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3 2020-06-24 15:49:35,897 - INFO [main:DatadirCleanupManager@79] - autopurge.purgeInterval set to 1 2020-06-24 15:49:35,898 - INFO [ PurgeTask:DatadirCleanupManager$PurgeTask@138] - Purge task started. 2020-06-24 15:49:35,899 - INFO [main:ManagedUtil@46] - Log4j found with jmx enabled. 2020-06-24 15:49:35,903 - INFO [PurgeTask:FileTxnSnapLog@103] - zookeeper.snapshot.trust.empty : false 2020-06-24 15:49:35,910 - INFO [ PurgeTask:DatadirCleanupManager$PurgeTask@144] - Purge task completed. 2020-06-24 15:49:35,975 - INFO [main:QuorumPeerMain@141] - Starting quorum peer 2020-06-24 15:49:35,983 - INFO [main:ServerCnxnFactory@135] - Using org.apache.zookeeper.server.NIOServerCnxnFactory as server connection factory 2020-06-24 15:49:35,986 - INFO [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 2 selector thread(s), 16 worker threads, and 64 kB direct buffers. 2020-06-24 15:49:35,992 - INFO [main:NIOServerCnxnFactory@686] - binding to port 0.0.0.0/0.0.0.0:10181 2020-06-24 15:49:35,994 - INFO [main:ServerCnxnFactory@135] - Using org.apache.zookeeper.server.NIOServerCnxnFactory as server connection factory 2020-06-24 15:49:35,995 - ERROR [main:QuorumPeerMain@101] - Unexpected exception, exiting abnormally java.lang.UnsupportedOperationException: SSL isn't supported in NIOServerCnxn at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:644) at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:155) at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:123) at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82) I have set the following properties in SERVER_JVMFLAGS in zkEnv.sh file : "-Dzookeeper.serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory -Dzookeeper.ssl.keyStore.location=/var/opt/vs/SecureInterface/keystore/CassSpkkeystore.p12 -Dzookeeper.ssl.keyStore.password=EvaiKiO1@123456 -Dzookeeper.ssl.trustStore.location=/var/opt/vs/SecureInterface/keystore/CassSpkTrustStore.jks -Dzookeeper.ssl.trustStore.password=EvaiKiO1@123456" Thanks, ----- Kuldeep Singh Budania On Mon, Jun 22, 2020 at 8:08 PM Jordan Zimmerman wrote: > It's the same as the normal ZooKeeper client: > https://zookeeper.apache.org/doc/r3.6.1/zookeeperAdmin.html#sc_authOptions > < > https://zookeeper.apache.org/doc/r3.6.1/zookeeperAdmin.html#sc_authOptions > > > > -Jordan > > > On Jun 22, 2020, at 5:50 AM, kuldeep singh > wrote: > > > > Hi Team, > > How we will do secure communication between the Curator framework and > > zookeeper 3.5.6 ? > > I didn't get any solution right now. > > I appreciate it if someone could help me with the same. > > > > Thanks, > > - > > Kuldeep Singh Budania > > Software Architect > > > > > > On Fri, Apr 17, 2020 at 4:53 PM Szalay-Bekő Máté < > szalay.beko.m...@gmail.com> > > wrote: > > > >> Hello Kuldeep, > >> > >> did you download the source from the ZooKeeper webpage, or checked out > from > >> git? > >> Anyway, the following commands should work: > >> > >> wget > >> > >> > https://downloads.apache.org/zookeeper/zookeeper-3.5.7/apache-zookeeper-3.5.7.tar.gz > >> tar xzvf ./apache-zookeeper-3.5.7.tar.gz > >> cd apache-zookeeper-3.5.7 > >> mvn clean install -DskipTests > >> > >> I tested it with OpenJDK 8u424 and maven 3.6.0. > >> > >> Kind regards, > >> Mate > >> > >> > >> On Fri, Apr 17, 2020 at 12:51 PM kuldeep singh < > kuldeep.sing...@gmail.com> > >> wrote: > >> > >>> Hi All, > >>> > >>> I have download the zookeeper source code and run the mvn command for > >>> packaging but get problem on "zookeeper-server" directory. > >>> > >>>
Re: upgrade from 3.4.5 to 3.5.6
Hi Team, How we will do secure communication between the Curator framework and zookeeper 3.5.6 ? I didn't get any solution right now. I appreciate it if someone could help me with the same. Thanks, - Kuldeep Singh Budania Software Architect On Fri, Apr 17, 2020 at 4:53 PM Szalay-Bekő Máté wrote: > Hello Kuldeep, > > did you download the source from the ZooKeeper webpage, or checked out from > git? > Anyway, the following commands should work: > > wget > > https://downloads.apache.org/zookeeper/zookeeper-3.5.7/apache-zookeeper-3.5.7.tar.gz > tar xzvf ./apache-zookeeper-3.5.7.tar.gz > cd apache-zookeeper-3.5.7 > mvn clean install -DskipTests > > I tested it with OpenJDK 8u424 and maven 3.6.0. > > Kind regards, > Mate > > > On Fri, Apr 17, 2020 at 12:51 PM kuldeep singh > wrote: > > > Hi All, > > > > I have download the zookeeper source code and run the mvn command for > > packaging but get problem on "zookeeper-server" directory. > > > > *command :- mvn clean package* > > > > [ERROR] Failed to execute goal > > org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile > > (default-compile) on project zookeeper: Fatal error compiling: > > java.lang.NullPointerException -> [Help 1] > > org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute > > goal org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile > > (default-compile) on project zookeeper: Fatal error compiling > > at > > > > > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:216) > > at > > > > > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153) > > at > > > > > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145) > > at > > > > > org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:108) > > at > > > > > org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:76) > > at > > > > > org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51) > > at > > > > > org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:116) > > at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:361) > > at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:155) > > at org.apache.maven.cli.MavenCli.execute(MavenCli.java:584) > > at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:213) > > at org.apache.maven.cli.MavenCli.main(MavenCli.java:157) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > > > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > at > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:498) > > at > > > > > org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289) > > at > > > org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229) > > at > > > > > org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415) > > at > > org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356) > > Caused by: org.apache.maven.plugin.MojoExecutionException: Fatal error > > compiling > > at > > > > > org.apache.maven.plugin.compiler.AbstractCompilerMojo.execute(AbstractCompilerMojo.java:1140) > > at > > > > > org.apache.maven.plugin.compiler.CompilerMojo.execute(CompilerMojo.java:188) > > at > > > > > org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:133) > > at > > > > > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208) > > ... 19 more > > Caused by: org.codehaus.plexus.compiler.CompilerException: > > java.lang.NullPointerException > > at > > > > > org.codehaus.plexus.compiler.javac.JavaxToolsCompiler.compileInProcess(JavaxToolsCompiler.java:173) > > at > > > > > org.codehaus.plexus.compiler.javac.JavacCompiler.performCompile(JavacCompiler.java:174) > > at > > >
Re: upgrade from 3.4.5 to 3.5.6
Hi, I appreciate if I will get a response as soon as possible, as I am stuck at this point. Thanks, - Kuldeep Singh Budania Software Architect On Wed, Jun 24, 2020 at 10:47 PM kuldeep singh wrote: > Hi, > > I got below error while setting SSL properties in zkEnv.sh > > > > > == > > 2020-06-24 15:49:35,864 - INFO [main:QuorumPeerConfig@133] - Reading > configuration from: /etc/zookeeper/zoo.cfg > > 2020-06-24 15:49:35,874 - INFO [main:QuorumPeerConfig@385] - > clientPortAddress is 0.0.0.0/0.0.0.0:10181 > > 2020-06-24 15:49:35,874 - INFO [main:QuorumPeerConfig@399] - > secureClientPortAddress is 0.0.0.0/0.0.0.0:2281 > > 2020-06-24 15:49:35,878 - INFO [main:X509Util@79] - Setting -D > jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated > TLS renegotiation > > 2020-06-24 15:49:35,897 - INFO [main:DatadirCleanupManager@78] - > autopurge.snapRetainCount set to 3 > > 2020-06-24 15:49:35,897 - INFO [main:DatadirCleanupManager@79] - > autopurge.purgeInterval set to 1 > > 2020-06-24 15:49:35,898 - INFO [ > PurgeTask:DatadirCleanupManager$PurgeTask@138] - Purge task started. > > 2020-06-24 15:49:35,899 - INFO [main:ManagedUtil@46] - Log4j found with > jmx enabled. > > 2020-06-24 15:49:35,903 - INFO [PurgeTask:FileTxnSnapLog@103] - > zookeeper.snapshot.trust.empty : false > > 2020-06-24 15:49:35,910 - INFO [ > PurgeTask:DatadirCleanupManager$PurgeTask@144] - Purge task completed. > > 2020-06-24 15:49:35,975 - INFO [main:QuorumPeerMain@141] - Starting > quorum peer > > 2020-06-24 15:49:35,983 - INFO [main:ServerCnxnFactory@135] - Using > org.apache.zookeeper.server.NIOServerCnxnFactory as server connection > factory > > 2020-06-24 15:49:35,986 - INFO [main:NIOServerCnxnFactory@673] - > Configuring NIO connection handler with 10s sessionless connection timeout, > 2 selector thread(s), 16 worker threads, and 64 kB direct buffers. > > 2020-06-24 15:49:35,992 - INFO [main:NIOServerCnxnFactory@686] - binding > to port 0.0.0.0/0.0.0.0:10181 > > 2020-06-24 15:49:35,994 - INFO [main:ServerCnxnFactory@135] - Using > org.apache.zookeeper.server.NIOServerCnxnFactory as server connection > factory > > 2020-06-24 15:49:35,995 - ERROR [main:QuorumPeerMain@101] - Unexpected > exception, exiting abnormally > > java.lang.UnsupportedOperationException: SSL isn't supported in > NIOServerCnxn > > at > org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:644) > > at > org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:155) > > at > org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:123) > > at > org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82) > > > > > > I have set the following properties in SERVER_JVMFLAGS in zkEnv.sh file : > "-Dzookeeper.serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory > -Dzookeeper.ssl.keyStore.location=/var/opt/vs/SecureInterface/keystore/CassSpkkeystore.p12 > -Dzookeeper.ssl.keyStore.password=EvaiKiO1@123456 > -Dzookeeper.ssl.trustStore.location=/var/opt/vs/SecureInterface/keystore/CassSpkTrustStore.jks > -Dzookeeper.ssl.trustStore.password=EvaiKiO1@123456" > > Thanks, > - > Kuldeep Singh Budania > > > > On Mon, Jun 22, 2020 at 8:08 PM Jordan Zimmerman < > jor...@jordanzimmerman.com> wrote: > >> It's the same as the normal ZooKeeper client: >> https://zookeeper.apache.org/doc/r3.6.1/zookeeperAdmin.html#sc_authOptions >> < >> https://zookeeper.apache.org/doc/r3.6.1/zookeeperAdmin.html#sc_authOptions >> > >> >> -Jordan >> >> > On Jun 22, 2020, at 5:50 AM, kuldeep singh >> wrote: >> > >> > Hi Team, >> > How we will do secure communication between the Curator framework and >> > zookeeper 3.5.6 ? >> > I didn't get any solution right now. >> > I appreciate it if someone could help me with the same. >> > >> > Thanks, >> > - >> > Kuldeep Singh Budania >> > Software Architect >> > >> > >> > On Fri, Apr 17, 2020 at 4:53 PM Szalay-Bekő Máté < >> szalay.beko.m...@gmail.com> >> > wrote: >> > >> >> Hello Kuldeep, >> >> >> >> did you download the source from the ZooKeeper webpage, or checked out >> from >> >> git? >> >> Anyway, the following commands should work: &g
