> My ZK server is up and running in secure mode What is your goal? You want to setup client-server SSL connection?
see: https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZooKeeper+SSL+User+Guide (or you want to have both SSL and SASL enabled?) Anyway, please remove the following line from the SERVER_JVMFLAGS: -Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty This is a configuration that makes sense only for the ZooKeeper client, not for the server. For the server, use the following: -Dzookeeper.serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory If it doesn't solve the issue, then can you please send your zoo.cfg file? Also can you please send the zkCli command you execute? (you need to connect to the secure ZooKeeper port, unless portUnification is enabled) Kind regards, Mate On Wed, Jul 1, 2020 at 9:48 AM kuldeep singh <kuldeep.sing...@gmail.com> wrote: > Hi, > > we have done below changes in java.env file > > export SERVER_JVMFLAGS=" > > -Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty > -Dzookeeper.ssl.keyStore.location=/root/zookeeper/ssl/testKeyStore.jks > -Dzookeeper.ssl.keyStore.password=testpass > -Dzookeeper.ssl.trustStore.location=/root/zookeeper/ssl/testTrustStore.jks > -Dzookeeper.ssl.trustStore.password=testpass" > > > > export CLIENT_JVMFLAGS=" > > -Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty > -Dzookeeper.client.secure=true > -Dzookeeper.ssl.keyStore.location=/root/zookeeper/ssl/testKeyStore.jks > -Dzookeeper.ssl.keyStore.password=testpass > -Dzookeeper.ssl.trustStore.location=/root/zookeeper/ssl/testTrustStore.jks > -Dzookeeper.ssl.trustStore.password=testpass" > > I have started the ZK server and it is up without any issue. > > But now when I login to ZkCli then it gives the below error. > > WatchedEvent state:AuthFailed type:None path:null > > > Zookeeper logs :- *2020-07-01 07:38:09,342 - WARN > [nioEventLoopGroup-4-2:ZooKeeperServer@1119] - No authentication provider > for scheme: ztpasswd has x509 ip digest* > > Please help me on this issue > > Thanks, > --------------------- > Kuldeep Singh Budania > Software Architect > > > On Wed, Jul 1, 2020 at 12:05 PM kuldeep singh <kuldeep.sing...@gmail.com> > wrote: > > > Hi, > > > > My ZK server is up and running in secure mode, But When I am trying to > > connect to the ZK server using ZKCli, it gives the below error. > > > > WatchedEvent state:AuthFailed type:None path:null > > > > > > Zookeeper logs :- *2020-07-01 07:38:09,342 - WARN > > [nioEventLoopGroup-4-2:ZooKeeperServer@1119] - No authentication > provider > > for scheme: ztpasswd has x509 ip digest* > > > > Can someone please help me on this issue. we are using the 3.5.6 version. > > > > I appreciate if I will get a response as soon as possible, as I am stuck > > at this point. > > > > Thanks, > > --------------------- > > Kuldeep Singh Budania > > Software Architect > > > > > > > > On Thu, Jun 25, 2020 at 11:54 AM Enrico Olivelli - Diennea > > <enrico.olive...@diennea.com.invalid> wrote: > > > >> I mean in zoo.cfg > >> Not as a system property > >> > >> Enrico > >> > >> Il giorno 25/06/20, 08:19 "Enrico Olivelli - Diennea" < > >> enrico.olive...@diennea.com.INVALID> ha scritto: > >> > >> Hi > >> You have to enable Netty on the server side > >> > >> Something like: > >> serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory > >> > >> Hope that helps > >> Enrico > >> > >> Il giorno 24/06/20, 19:17 "kuldeep singh" < > kuldeep.sing...@gmail.com> > >> ha scritto: > >> > >> Hi, > >> > >> I got below error while setting SSL properties in zkEnv.sh > >> > >> > >> > >> > >> > ========================================================================================== > >> > >> 2020-06-24 15:49:35,864 - INFO [main:QuorumPeerConfig@133] - > >> Reading > >> configuration from: /etc/zookeeper/zoo.cfg > >> > >> 2020-06-24 15:49:35,874 - INFO [main:QuorumPeerConfig@385] - > >> clientPortAddress is 0.0.0.0/0.0.0.0:10181 > >> > >> 2020-06-24 15:49:35,874 - INFO [main:QuorumPeerConfig@399] - > >> secureClientPortAddress is 0.0.0.0/0.0.0.0:2281 > >> > >> 2020-06-24 15:49:35,878 - INFO [main:X509Util@79] - Setting -D > >> jdk.tls.rejectClientInitiatedRenegotiation=true to disable > >> client-initiated > >> TLS renegotiation > >> > >> 2020-06-24 15:49:35,897 - INFO [main:DatadirCleanupManager@78] > - > >> autopurge.snapRetainCount set to 3 > >> > >> 2020-06-24 15:49:35,897 - INFO [main:DatadirCleanupManager@79] > - > >> autopurge.purgeInterval set to 1 > >> > >> 2020-06-24 15:49:35,898 - INFO [ > >> PurgeTask:DatadirCleanupManager$PurgeTask@138] - Purge task > >> started. > >> > >> 2020-06-24 15:49:35,899 - INFO [main:ManagedUtil@46] - Log4j > >> found with > >> jmx enabled. > >> > >> 2020-06-24 15:49:35,903 - INFO [PurgeTask:FileTxnSnapLog@103] > - > >> zookeeper.snapshot.trust.empty : false > >> > >> 2020-06-24 15:49:35,910 - INFO [ > >> PurgeTask:DatadirCleanupManager$PurgeTask@144] - Purge task > >> completed. > >> > >> 2020-06-24 15:49:35,975 - INFO [main:QuorumPeerMain@141] - > >> Starting quorum > >> peer > >> > >> 2020-06-24 15:49:35,983 - INFO [main:ServerCnxnFactory@135] - > >> Using > >> org.apache.zookeeper.server.NIOServerCnxnFactory as server > >> connection > >> factory > >> > >> 2020-06-24 15:49:35,986 - INFO [main:NIOServerCnxnFactory@673] > - > >> Configuring NIO connection handler with 10s sessionless > >> connection timeout, > >> 2 selector thread(s), 16 worker threads, and 64 kB direct > buffers. > >> > >> 2020-06-24 15:49:35,992 - INFO [main:NIOServerCnxnFactory@686] > >> - binding > >> to port 0.0.0.0/0.0.0.0:10181 > >> > >> 2020-06-24 15:49:35,994 - INFO [main:ServerCnxnFactory@135] - > >> Using > >> org.apache.zookeeper.server.NIOServerCnxnFactory as server > >> connection > >> factory > >> > >> 2020-06-24 15:49:35,995 - ERROR [main:QuorumPeerMain@101] - > >> Unexpected > >> exception, exiting abnormally > >> > >> java.lang.UnsupportedOperationException: SSL isn't supported in > >> NIOServerCnxn > >> > >> at > >> > >> > org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:644) > >> > >> at > >> > >> > org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:155) > >> > >> at > >> > >> > org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:123) > >> > >> at > >> > >> > org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82) > >> > >> > >> > >> > >> > >> I have set the following properties in SERVER_JVMFLAGS in > >> zkEnv.sh file : > >> > >> > "-Dzookeeper.serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory > >> > >> > -Dzookeeper.ssl.keyStore.location=/var/opt/vs/SecureInterface/keystore/CassSpkkeystore.p12 > >> -Dzookeeper.ssl.keyStore.password=EvaiKiO1@123456 > >> > >> > -Dzookeeper.ssl.trustStore.location=/var/opt/vs/SecureInterface/keystore/CassSpkTrustStore.jks > >> -Dzookeeper.ssl.trustStore.password=EvaiKiO1@123456" > >> > >> Thanks, > >> --------------------- > >> Kuldeep Singh Budania > >> > >> > >> > >> On Mon, Jun 22, 2020 at 8:08 PM Jordan Zimmerman < > >> jor...@jordanzimmerman.com> > >> wrote: > >> > >> > It's the same as the normal ZooKeeper client: > >> > > >> > https://zookeeper.apache.org/doc/r3.6.1/zookeeperAdmin.html#sc_authOptions > >> > < > >> > > >> > https://zookeeper.apache.org/doc/r3.6.1/zookeeperAdmin.html#sc_authOptions > >> > > > >> > > >> > -Jordan > >> > > >> > > On Jun 22, 2020, at 5:50 AM, kuldeep singh < > >> kuldeep.sing...@gmail.com> > >> > wrote: > >> > > > >> > > Hi Team, > >> > > How we will do secure communication between the Curator > >> framework and > >> > > zookeeper 3.5.6 ? > >> > > I didn't get any solution right now. > >> > > I appreciate it if someone could help me with the same. > >> > > > >> > > Thanks, > >> > > --------------------- > >> > > Kuldeep Singh Budania > >> > > Software Architect > >> > > > >> > > > >> > > On Fri, Apr 17, 2020 at 4:53 PM Szalay-Bekő Máté < > >> > szalay.beko.m...@gmail.com> > >> > > wrote: > >> > > > >> > >> Hello Kuldeep, > >> > >> > >> > >> did you download the source from the ZooKeeper webpage, or > >> checked out > >> > from > >> > >> git? > >> > >> Anyway, the following commands should work: > >> > >> > >> > >> wget > >> > >> > >> > >> > >> > > >> > https://downloads.apache.org/zookeeper/zookeeper-3.5.7/apache-zookeeper-3.5.7.tar.gz > >> > >> tar xzvf ./apache-zookeeper-3.5.7.tar.gz > >> > >> cd apache-zookeeper-3.5.7 > >> > >> mvn clean install -DskipTests > >> > >> > >> > >> I tested it with OpenJDK 8u424 and maven 3.6.0. > >> > >> > >> > >> Kind regards, > >> > >> Mate > >> > >> > >> > >> > >> > >> On Fri, Apr 17, 2020 at 12:51 PM kuldeep singh < > >> > kuldeep.sing...@gmail.com> > >> > >> wrote: > >> > >> > >> > >>> Hi All, > >> > >>> > >> > >>> I have download the zookeeper source code and run the mvn > >> command for > >> > >>> packaging but get problem on "zookeeper-server" directory. > >> > >>> > >> > >>> *command :- mvn clean package* > >> > >>> > >> > >>> [ERROR] Failed to execute goal > >> > >>> > org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile > >> > >>> (default-compile) on project zookeeper: Fatal error > >> compiling: > >> > >>> java.lang.NullPointerException -> [Help 1] > >> > >>> org.apache.maven.lifecycle.LifecycleExecutionException: > >> Failed to > >> > execute > >> > >>> goal > >> org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile > >> > >>> (default-compile) on project zookeeper: Fatal error > >> compiling > >> > >>> at > >> > >>> > >> > >>> > >> > >> > >> > > >> > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:216) > >> > >>> at > >> > >>> > >> > >>> > >> > >> > >> > > >> > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153) > >> > >>> at > >> > >>> > >> > >>> > >> > >> > >> > > >> > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145) > >> > >>> at > >> > >>> > >> > >>> > >> > >> > >> > > >> > org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:108) > >> > >>> at > >> > >>> > >> > >>> > >> > >> > >> > > >> > org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:76) > >> > >>> at > >> > >>> > >> > >>> > >> > >> > >> > > >> > org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51) > >> > >>> at > >> > >>> > >> > >>> > >> > >> > >> > > >> > org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:116) > >> > >>> at > >> > org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:361) > >> > >>> at > >> org.apache.maven.DefaultMaven.execute(DefaultMaven.java:155) > >> > >>> at > >> org.apache.maven.cli.MavenCli.execute(MavenCli.java:584) > >> > >>> at > >> org.apache.maven.cli.MavenCli.doMain(MavenCli.java:213) > >> > >>> at > >> org.apache.maven.cli.MavenCli.main(MavenCli.java:157) > >> > >>> at > >> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > >> > >>> at > >> > >>> > >> > >>> > >> > >> > >> > > >> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > >> > >>> at > >> > >>> > >> > >>> > >> > >> > >> > > >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >> > >>> at java.lang.reflect.Method.invoke(Method.java:498) > >> > >>> at > >> > >>> > >> > >>> > >> > >> > >> > > >> > org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289) > >> > >>> at > >> > >>> > >> > >> > >> > > >> > org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229) > >> > >>> at > >> > >>> > >> > >>> > >> > >> > >> > > >> > org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415) > >> > >>> at > >> > >>> > >> > > >> > org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356) > >> > >>> Caused by: org.apache.maven.plugin.MojoExecutionException: > >> Fatal error > >> > >>> compiling > >> > >>> at > >> > >>> > >> > >>> > >> > >> > >> > > >> > org.apache.maven.plugin.compiler.AbstractCompilerMojo.execute(AbstractCompilerMojo.java:1140) > >> > >>> at > >> > >>> > >> > >>> > >> > >> > >> > > >> > org.apache.maven.plugin.compiler.CompilerMojo.execute(CompilerMojo.java:188) > >> > >>> at > >> > >>> > >> > >>> > >> > >> > >> > > >> > org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:133) > >> > >>> at > >> > >>> > >> > >>> > >> > >> > >> > > >> > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208) > >> > >>> ... 19 more > >> > >>> Caused by: org.codehaus.plexus.compiler.CompilerException: > >> > >>> java.lang.NullPointerException > >> > >>> at > >> > >>> > >> > >>> > >> > >> > >> > > >> > org.codehaus.plexus.compiler.javac.JavaxToolsCompiler.compileInProcess(JavaxToolsCompiler.java:173) > >> > >>> at > >> > >>> > >> > >>> > >> > >> > >> > > >> > org.codehaus.plexus.compiler.javac.JavacCompiler.performCompile(JavacCompiler.java:174) > >> > >>> at > >> > >>> > >> > >>> > >> > >> > >> > > >> > org.apache.maven.plugin.compiler.AbstractCompilerMojo.execute(AbstractCompilerMojo.java:1129) > >> > >>> ... 22 more > >> > >>> Caused by: java.lang.RuntimeException: > >> java.lang.NullPointerException > >> > >>> at > >> com.sun.tools.javac.main.Main.compile(Main.java:559) > >> > >>> at > >> > >>> > >> com.sun.tools.javac.api.JavacTaskImpl.doCall(JavacTaskImpl.java:129) > >> > >>> at > >> > >>> > >> com.sun.tools.javac.api.JavacTaskImpl.call(JavacTaskImpl.java:138) > >> > >>> at > >> > >>> > >> > >>> > >> > >> > >> > > >> > org.codehaus.plexus.compiler.javac.JavaxToolsCompiler.compileInProcess(JavaxToolsCompiler.java:126) > >> > >>> ... 24 more > >> > >>> Caused by: java.lang.NullPointerException > >> > >>> at > >> > >>> > >> > >>> > >> > >> > >> > > >> > com.sun.tools.javac.model.JavacTypes.getOverriddenMethods(JavacTypes.java:306) > >> > >>> at > com.sun.tools.doclint.Env.setCurrent(Env.java:155) > >> > >>> at > >> com.sun.tools.doclint.Checker.scan(Checker.java:144) > >> > >>> at > >> com.sun.tools.doclint.DocLint$2.visitDecl(DocLint.java:285) > >> > >>> at > >> > >>> > >> com.sun.tools.doclint.DocLint$DeclScanner.visitClass(DocLint.java:359) > >> > >>> at > >> > >>> > >> com.sun.tools.doclint.DocLint$DeclScanner.visitClass(DocLint.java:346) > >> > >>> at > >> > >>> > >> com.sun.tools.javac.tree.JCTree$JCClassDecl.accept(JCTree.java:720) > >> > >>> at > >> > >>> > >> com.sun.source.util.TreePathScanner.scan(TreePathScanner.java:68) > >> > >>> at > >> > >>> > >> com.sun.source.util.TreeScanner.scanAndReduce(TreeScanner.java:81) > >> > >>> at > >> > >>> > >> com.sun.source.util.TreeScanner.visitNewClass(TreeScanner.java:280) > >> > >>> at > >> > >>> > >> com.sun.tools.javac.tree.JCTree$JCNewClass.accept(JCTree.java:1532) > >> > >>> at > >> > >>> > >> com.sun.source.util.TreePathScanner.scan(TreePathScanner.java:68) > >> > >>> at > >> > >>> > >> com.sun.source.util.TreeScanner.scanAndReduce(TreeScanner.java:81) > >> > >>> at > >> > >>> > >> com.sun.source.util.TreeScanner.visitVariable(TreeScanner.java:153) > >> > >>> at > >> > >>> > >> > > >> > com.sun.tools.doclint.DocLint$DeclScanner.visitVariable(DocLint.java:373) > >> > >>> at > >> > >>> > >> > > >> > com.sun.tools.doclint.DocLint$DeclScanner.visitVariable(DocLint.java:346) > >> > >>> at > >> > >>> > >> com.sun.tools.javac.tree.JCTree$JCVariableDecl.accept(JCTree.java:864) > >> > >>> at > >> > >>> > >> com.sun.source.util.TreePathScanner.scan(TreePathScanner.java:68) > >> > >>> at > >> > >>> > >> com.sun.source.util.TreeScanner.scanAndReduce(TreeScanner.java:81) > >> > >>> at > >> com.sun.source.util.TreeScanner.scan(TreeScanner.java:91) > >> > >>> at > >> > >>> > >> com.sun.source.util.TreeScanner.scanAndReduce(TreeScanner.java:99) > >> > >>> at > >> > >> > >> com.sun.source.util.TreeScanner.visitClass(TreeScanner.java:133) > >> > >>> at > >> > >>> > >> com.sun.tools.doclint.DocLint$DeclScanner.visitClass(DocLint.java:360) > >> > >>> at > >> > >>> > >> com.sun.tools.doclint.DocLint$DeclScanner.visitClass(DocLint.java:346) > >> > >>> at > >> > >>> > >> com.sun.tools.javac.tree.JCTree$JCClassDecl.accept(JCTree.java:720) > >> > >>> at > >> > >>> > >> com.sun.source.util.TreePathScanner.scan(TreePathScanner.java:68) > >> > >>> at > >> com.sun.source.util.TreeScanner.scan(TreeScanner.java:91) > >> > >>> at > >> > >>> > >> com.sun.source.util.TreeScanner.scanAndReduce(TreeScanner.java:99) > >> > >>> at > >> > >>> > >> > >> > >> > > >> > com.sun.source.util.TreeScanner.visitCompilationUnit(TreeScanner.java:120) > >> > >>> at > >> > >>> > >> > >>> > >> > >> > >> > > >> > com.sun.tools.doclint.DocLint$DeclScanner.visitCompilationUnit(DocLint.java:354) > >> > >>> at > >> > >>> > >> > >>> > >> > >> > >> > > >> > com.sun.tools.doclint.DocLint$DeclScanner.visitCompilationUnit(DocLint.java:346) > >> > >>> at > >> > >>> > >> > > >> > com.sun.tools.javac.tree.JCTree$JCCompilationUnit.accept(JCTree.java:550) > >> > >>> at > >> > >>> > >> com.sun.source.util.TreePathScanner.scan(TreePathScanner.java:68) > >> > >>> at > >> com.sun.tools.doclint.DocLint$3.started(DocLint.java:296) > >> > >>> at > >> > >>> > >> > >>> > >> > >> > >> > > >> > com.sun.tools.javac.api.ClientCodeWrapper$WrappedTaskListener.started(ClientCodeWrapper.java:668) > >> > >>> at > >> > >>> > >> > >>> > >> > >> > >> > > >> > com.sun.tools.javac.api.MultiTaskListener.started(MultiTaskListener.java:103) > >> > >>> at > >> > >>> > >> com.sun.tools.javac.main.JavaCompiler.attribute(JavaCompiler.java:1240) > >> > >>> at > >> > >>> > >> com.sun.tools.javac.main.JavaCompiler.compile2(JavaCompiler.java:901) > >> > >>> at > >> > >>> > >> com.sun.tools.javac.main.JavaCompiler.compile(JavaCompiler.java:860) > >> > >>> at > >> com.sun.tools.javac.main.Main.compile(Main.java:523) > >> > >>> ... 27 more > >> > >>> [ERROR] > >> > >>> [ERROR] Re-run Maven using the -X switch to enable full > >> debug logging. > >> > >>> [ERROR] > >> > >>> [ERROR] For more information about the errors and possible > >> solutions, > >> > >>> please read the following articles: > >> > >>> [ERROR] [Help 1] > >> > >>> > >> > > >> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException > >> > >>> > >> > >>> Can someone help me on this issue? > >> > >>> > >> > >>> Thanks, > >> > >>> --------------------- > >> > >>> Kuldeep Singh Budania > >> > >>> > >> > >>> > >> > >>> > >> > >>> On Sat, Apr 4, 2020 at 5:57 PM Szalay-Bekő Máté < > >> > >>> szalay.beko.m...@gmail.com> > >> > >>> wrote: > >> > >>> > >> > >>>> these exceptions can mean many things... I think this can > >> be even > >> > >> normal > >> > >>>> duding rolling restart (as some connections get broken in > >> this case) > >> > >>>> > >> > >>>> However, I saw cases already when exceptions like these > >> killed > >> > receiver > >> > >>> or > >> > >>>> sender threads in QuorumCnxManager / Leader Election in > >> such a way > >> > that > >> > >>>> they were not able to recover, so the node was unable to > >> connect to > >> > any > >> > >>>> quorum until restart. I remember seeing this in 3.4 too. > >> > >>>> > >> > >>>> Do you see these exceptions in the second server (the one > >> which you > >> > >> just > >> > >>>> upgraded in step 3)? > >> > >>>> Is this issue reproducible? > >> > >>>> > >> > >>>> What is the tickTime and initLimit you use? Maybe the > >> server just > >> > >> require > >> > >>>> more time to sync? > >> > >>>> > >> > >>>> I would need more logs to really see what happened. Can > >> you create a > >> > >> Jira > >> > >>>> issue and upload the logs and also the ZooKeeper configs? > >> I am happy > >> > to > >> > >>>> take a closer look. > >> > >>>> (if you need to re-run the test to collect the logs, then > >> enabling > >> > >> DEBUG > >> > >>>> logs would be great. The INFO level logs are usually > >> enough for these > >> > >>>> problems, but one can never know...) > >> > >>>> > >> > >>>> Kind regards, > >> > >>>> Mate > >> > >>>> > >> > >>>> > >> > >>>> On Fri, Apr 3, 2020 at 10:05 AM kuldeep singh < > >> > >> kuldeep.sing...@gmail.com > >> > >>>> > >> > >>>> wrote: > >> > >>>> > >> > >>>>> Hi Team, > >> > >>>>> > >> > >>>>> I have done some POC on rolling upgrade and found below > >> result. > >> > >>>>> > >> > >>>>> > >> > >>>>> 1. On 1st node upgrade zookeeper . Traffic was running > >> fine > >> > >> because > >> > >>> 2 > >> > >>>>> nodes are already on old zookeeper. > >> > >>>>> 2. On 1st node upgrade our application and didn’t find > >> any issue > >> > >>>>> 3. On 2nd node upgrade zookeeper but got below error > >> and zookeeper > >> > >>> is > >> > >>>>> not taking any requests > >> > >>>>> 4. > >> > >>>>> > >> > >>>>> java.io.EOFException > >> > >>>>> > >> > >>>>> at > >> java.io.DataInputStream.readInt(DataInputStream.java:392) > >> > >>>>> > >> > >>>>> at > >> > >>>>> > >> > >>>>> > >> > >>>> > >> > >>> > >> > >> > >> > > >> > org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:747) > >> > >>>>> > >> > >>>>> 2020-03-30 14:19:55,587 - WARN > >> > >>>>> [RecvWorker:1:QuorumCnxManager$RecvWorker@765] - > >> Interrupting > >> > >>> SendWorker > >> > >>>>> > >> > >>>>> 2020-03-30 14:19:55,588 - ERROR [LearnerHandler-/ > >> 192.168.44.73:33754 > >> > >>>>> :LearnerHandler@562] - Unexpected exception causing > >> shutdown while > >> > >>> sock > >> > >>>>> still open > >> > >>>>> > >> > >>>>> java.io.EOFException > >> > >>>>> > >> > >>>>> at > >> java.io.DataInputStream.readInt(DataInputStream.java:392) > >> > >>>>> > >> > >>>>> at > >> > >>>>> > >> > >> > >> org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63) > >> > >>>>> > >> > >>>>> at > >> > >>>>> > >> > >>>>> > >> > >>>> > >> > >>> > >> > >> > >> > > >> > org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:83) > >> > >>>>> > >> > >>>>> at > >> > >>>>> > >> > >>>> > >> > >>> > >> > >> > >> > > >> > org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:108) > >> > >>>>> > >> > >>>>> at > >> > >>>>> > >> > >>>>> > >> > >>>> > >> > >>> > >> > >> > >> > > >> > org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:476) > >> > >>>>> > >> > >>>>> 2020-03-30 14:19:55,588 - WARN > >> > >>>>> [SendWorker:1:QuorumCnxManager$SendWorker@679] - > >> Interrupted while > >> > >>>> waiting > >> > >>>>> for message on queue > >> > >>>>> > >> > >>>>> Please let me know is this the known issue or this is > >> different issue > >> > >>>> which > >> > >>>>> is mention in Apache zookeeper documentation when > >> upgrading from > >> > >> 3.4.5 > >> > >>> to > >> > >>>>> 3.5.6 > >> > >>>>> > >> > >>>>> Thanks, > >> > >>>>> --------------------- > >> > >>>>> Kuldeep Singh Budania > >> > >>>>> Software Architect > >> > >>>>> > >> > >>>>> > >> > >>>>> > >> > >>>>> On Sun, Mar 29, 2020 at 9:06 AM Alexander Shraer < > >> shra...@gmail.com> > >> > >>>>> wrote: > >> > >>>>> > >> > >>>>>> +1 to what Mate said (I wrote the quoted instructions). > >> > >>>>>> > >> > >>>>>> > >> > >>>>>> > >> > >>>>>> On Tue, Mar 24, 2020 at 7:03 AM Szalay-Bekő Máté < > >> > >>>>>> szalay.beko.m...@gmail.com> > >> > >>>>>> wrote: > >> > >>>>>> > >> > >>>>>>> Hi Kuldeep, > >> > >>>>>>> > >> > >>>>>>> I just want to provide you some background info about > >> our > >> > >>>>> documentation. > >> > >>>>>>> The reason to upgrade to 3.4.6 first is to avoid the > >> following > >> > >>> error: > >> > >>>>>>> > >> > >>>>>>>> 2013-01-30 11:32:10,663 [myid:2] - WARN [localhost/ > >> > >>> 127.0.0.1:2784 > >> > >>>>>>> :QuorumCnxManager@349] - Invalid server id: -65536 > >> > >>>>>>> > >> > >>>>>>> This error comes because of the protocol changes > between > >> > >> ZooKeeper > >> > >>>>> server > >> > >>>>>>> nodes during connection initiation for leader > election. > >> In > >> > >>> ZooKeeper > >> > >>>>> 3.5 > >> > >>>>>> a > >> > >>>>>>> protocol version was introduced (see ZOOKEEPER-107) > and > >> since > >> > >> that > >> > >>>> time > >> > >>>>>> the > >> > >>>>>>> fist long value sent in the initial message is not the > >> server ID > >> > >>> but > >> > >>>>> the > >> > >>>>>>> protocol version (-65536). In ZooKeeper 3.4.6 we made > >> the old 3.4 > >> > >>>>>>> ZooKeepers backward compatible, so they are able to > >> parse both > >> > >> the > >> > >>>> old > >> > >>>>>> and > >> > >>>>>>> the new protocol format (see ZOOKEEPER-1633). This > >> issue happens > >> > >>> only > >> > >>>>>> when > >> > >>>>>>> you need to use old (3.4.0 - 3.4.5) and new (3.5.0+) > >> ZooKeeper > >> > >>>> servers > >> > >>>>>>> together in the same cluster. During a rolling > upgrade, > >> this is > >> > >>>> usually > >> > >>>>>> the > >> > >>>>>>> case to have old and new ZooKeepers present together. > >> > >>>>>>> > >> > >>>>>>> The fact that you haven't seen any issues might be > >> caused by the > >> > >>>> order > >> > >>>>> of > >> > >>>>>>> the servers. In ZooKeeper the connection initiation > >> between the > >> > >>>> servers > >> > >>>>>>> during the leader election follows a specific rule. As > >> far as I > >> > >>>>> remember > >> > >>>>>>> always the server with the larger ID 'wins the > >> challenge', so it > >> > >> is > >> > >>>>>>> possible, that the old server didn't need to parse any > >> initial > >> > >>>> message > >> > >>>>>> (if > >> > >>>>>>> it had the largest ID) and this is why you haven't > seen > >> the > >> > >> issue. > >> > >>>> Also > >> > >>>>>>> having 2 nodes up from the 3 nodes cluster still makes > >> the > >> > >> cluster > >> > >>>> work > >> > >>>>>> (so > >> > >>>>>>> you should also check if all the servers are part of > >> the quorum). > >> > >>>>>>> > >> > >>>>>>> I agree with Enrico and Norbert, the safest and most > >> stable way > >> > >> is > >> > >>>>>> upgrade > >> > >>>>>>> first to 3.4.latest, then go to 3.5.latest. Still, if > >> you don't > >> > >> see > >> > >>>>> that > >> > >>>>>>> you would hit this specific issue (e.g. no "Invalid > >> server id" in > >> > >>> the > >> > >>>>> log > >> > >>>>>>> files), and all the three servers can handle traffic, > >> then maybe > >> > >>> you > >> > >>>>>> don't > >> > >>>>>>> need to upgrade first to 3.4.latest, it is your > >> decision. > >> > >>> Definitely > >> > >>>>> you > >> > >>>>>>> should test it first, as suggested by the others. > >> > >>>>>>> > >> > >>>>>>> Kind regards, > >> > >>>>>>> Mate > >> > >>>>>>> > >> > >>>>>>> On Tue, Mar 24, 2020 at 12:29 PM Norbert Kalmar > >> > >>>>>>> <nkal...@cloudera.com.invalid> wrote: > >> > >>>>>>> > >> > >>>>>>>> Hi, > >> > >>>>>>>> > >> > >>>>>>>> That guide is to upgrade to 3.5.0, which was an alpha > >> version. > >> > >> A > >> > >>>> lot > >> > >>>>>> has > >> > >>>>>>>> changed for the first stable release of 3.5.5 and > then > >> a few > >> > >>> more, > >> > >>>>> even > >> > >>>>>>>> rolling upgrade issues have been fixed for 3.5.6. > >> > >>>>>>>> This is a more up-to-date guide: > >> > >>>>>>>> > >> > >>> > >> https://cwiki.apache.org/confluence/display/ZOOKEEPER/Upgrade+FAQ > >> > >>>>>>>> > >> > >>>>>>>> If you have done your testing (with prod snapshot!), > >> then you > >> > >> can > >> > >>>>> skip > >> > >>>>>>> 3.4 > >> > >>>>>>>> latest upgrade, but keep in mind we do our > >> recommendations for > >> > >> a > >> > >>>>>> reason. > >> > >>>>>>>> There were issues reported and/or found during > >> testing. Some > >> > >> are > >> > >>>>> fixed > >> > >>>>>>> with > >> > >>>>>>>> 3.5.6, some only happens if certain conditions stand > >> > >>> (IOException: > >> > >>>> No > >> > >>>>>>>> snapshot found - mentioned in the guide, fixed in > >> 3.5.6). > >> > >>>>>>>> > >> > >>>>>>>> So it is up to you, I would still recommend to do an > >> 3.4 > >> > >> upgrade > >> > >>>>> first, > >> > >>>>>>> if > >> > >>>>>>>> it's feasible. > >> > >>>>>>>> > >> > >>>>>>>> Regards, > >> > >>>>>>>> Norbert > >> > >>>>>>>> > >> > >>>>>>>> On Tue, Mar 24, 2020 at 11:45 AM kuldeep singh < > >> > >>>>>>> kuldeep.sing...@gmail.com> > >> > >>>>>>>> wrote: > >> > >>>>>>>> > >> > >>>>>>>>> Hi, > >> > >>>>>>>>> > >> > >>>>>>>>> Current Zookeeper version :- 3.4.5 > >> > >>>>>>>>> Upgraded version :- 3.5.6 > >> > >>>>>>>>> > >> > >>>>>>>>> We are not going with 3.5.7. Our final decision is > >> zookeeper > >> > >>>>> version > >> > >>>>>> is > >> > >>>>>>>>> 3.5.6 > >> > >>>>>>>>> as per your reply first we need to move latest > >> version of > >> > >>> 3.4.x, > >> > >>>>> like > >> > >>>>>>>> below > >> > >>>>>>>>> > >> > >>>>>>>>> 3.4.5 -> 3.4.14 -> 3.5.6 (Correct me if I am wrong > >> here) > >> > >>>>>>>>> > >> > >>>>>>>>> But if We are not facing any problem that i have > >> shared you > >> > >>> that > >> > >>>> we > >> > >>>>>>> have > >> > >>>>>>>>> set up of 3 node cluster where 2 node are on 3.5.6 > >> version > >> > >> and > >> > >>> 1 > >> > >>>>> node > >> > >>>>>>> on > >> > >>>>>>>>> 3.4.5, Everything is running fine and didn't get any > >> issue, > >> > >> So > >> > >>>> what > >> > >>>>>>> other > >> > >>>>>>>>> problem we can face if we directly move to 3.5.6 > >> > >>>>>>>>> > >> > >>>>>>>>> Thanks, > >> > >>>>>>>>> --------------------- > >> > >>>>>>>>> Kuldeep Singh Budania > >> > >>>>>>>>> Software Architect > >> > >>>>>>>>> > >> > >>>>>>>>> > >> > >>>>>>>>> On Tue, Mar 24, 2020 at 3:58 PM Enrico Olivelli < > >> > >>>>> eolive...@gmail.com > >> > >>>>>>> > >> > >>>>>>>>> wrote: > >> > >>>>>>>>> > >> > >>>>>>>>>> Hi > >> > >>>>>>>>>> You have to upgrade to latest 3.4.x Zookeeper then > >> you will > >> > >>>>> upgrade > >> > >>>>>>> to > >> > >>>>>>>>>> 3.5.7. > >> > >>>>>>>>>> All should run well without issues > >> > >>>>>>>>>> > >> > >>>>>>>>>> > >> > >>>>>>>>>> Enrico > >> > >>>>>>>>>> > >> > >>>>>>>>>> Il Mar 24 Mar 2020, 10:18 kuldeep singh < > >> > >>>>> kuldeep.sing...@gmail.com > >> > >>>>>>> > >> > >>>>>>> ha > >> > >>>>>>>>>> scritto: > >> > >>>>>>>>>> > >> > >>>>>>>>>>> Hi Team, > >> > >>>>>>>>>>> > >> > >>>>>>>>>>> We are upgrading zookeeper from 3.4.5 to 3.5.6. I > >> have > >> > >> set > >> > >>>> up 3 > >> > >>>>>>> node > >> > >>>>>>>>>>> cluster where 2 node are on 3.5.6 version and 1 > >> node on > >> > >>>> 3.4.5. > >> > >>>>>>>>>>> > >> > >>>>>>>>>>> Everything is running fine and didn't get any > issue > >> on my > >> > >>>>> system. > >> > >>>>>>>>>>> > >> > >>>>>>>>>>> but I found something on apache site that first > we > >> need > >> > >> to > >> > >>>>>> upgrade > >> > >>>>>>>> on > >> > >>>>>>>>>>> 3.4.6 than we can upgrade to 3.5.6. So is it > >> mandatory > >> > >> to > >> > >>> go > >> > >>>>> on > >> > >>>>>>>> 3.4.6 > >> > >>>>>>>>>>> first. > >> > >>>>>>>>>>> > >> > >>>>>>>>>>> *Upgrading to 3.5.0* > >> > >>>>>>>>>>> > >> > >>>>>>>>>>> Upgrading a running ZooKeeper ensemble to 3.5.0 > >> should be > >> > >>>> done > >> > >>>>>> only > >> > >>>>>>>>> after > >> > >>>>>>>>>>> upgrading your ensemble to the 3.4.6 release. Note > >> that > >> > >>> this > >> > >>>> is > >> > >>>>>>> only > >> > >>>>>>>>>>> necessary for rolling upgrades (if you're fine > with > >> > >>> shutting > >> > >>>>> down > >> > >>>>>>> the > >> > >>>>>>>>>>> system completely, you don't have to go through > >> 3.4.6). > >> > >> If > >> > >>>> you > >> > >>>>>>>> attempt > >> > >>>>>>>>> a > >> > >>>>>>>>>>> rolling upgrade without going through 3.4.6 (for > >> example > >> > >>> from > >> > >>>>>>> 3.4.5), > >> > >>>>>>>>> you > >> > >>>>>>>>>>> may get the following error: > >> > >>>>>>>>>>> > >> > >>>>>>>>>>> 2013-01-30 11:32:10,663 [myid:2] - INFO > [localhost/ > >> > >>>>>> 127.0.0.1:2784 > >> > >>>>>>>>>>> :QuorumCnxManager$Listener@498] - Received > >> connection > >> > >>>> request > >> > >>>>> / > >> > >>>>>>>>>>> 127.0.0.1:60876 > >> > >>>>>>>>>>> > >> > >>>>>>>>>>> 2013-01-30 11:32:10,663 [myid:2] - WARN > [localhost/ > >> > >>>>>> 127.0.0.1:2784 > >> > >>>>>>>>>>> :QuorumCnxManager@349] - Invalid server id: > -65536 > >> > >>>>>>>>>>> > >> > >>>>>>>>>>> During a rolling upgrade, each server is taken > down > >> in > >> > >> turn > >> > >>>> and > >> > >>>>>>>>> rebooted > >> > >>>>>>>>>>> with the new 3.5.0 binaries. Before starting the > >> server > >> > >>> with > >> > >>>>>> 3.5.0 > >> > >>>>>>>>>>> binaries, we highly recommend updating the > >> configuration > >> > >>> file > >> > >>>>> so > >> > >>>>>>> that > >> > >>>>>>>>> all > >> > >>>>>>>>>>> server statements "server.x=..." contain client > >> ports > >> > >> (see > >> > >>>> the > >> > >>>>>>>> section > >> > >>>>>>>>>>> Specifying > >> > >>>>>>>>>>> the client port). As explained earlier you may > >> leave the > >> > >>>>>>>> configuration > >> > >>>>>>>>>> in a > >> > >>>>>>>>>>> single file, as well as leave the > >> > >>>> clientPort/clientPortAddress > >> > >>>>>>>>> statements > >> > >>>>>>>>>>> (although if you specify client ports in the new > >> format, > >> > >>>> these > >> > >>>>>>>>> statements > >> > >>>>>>>>>>> are now redundant). > >> > >>>>>>>>>>> > >> > >>>>>>>>>>> Could you please let me know about this case. > >> Appreciate > >> > >> if > >> > >>>>>> respond > >> > >>>>>>>>> soon. > >> > >>>>>>>>>>> > >> > >>>>>>>>>>> Thanks, > >> > >>>>>>>>>>> --------------------- > >> > >>>>>>>>>>> Kuldeep Singh Budania > >> > >>>>>>>>>>> > >> > >>>>>>>>>> > >> > >>>>>>>>> > >> > >>>>>>>> > >> > >>>>>>> > >> > >>>>>> > >> > >>>>> > >> > >>>> > >> > >>> > >> > >> > >> > > >> > > >> > >> > >> ________________________________ > >> > >> CONFIDENTIALITY & PRIVACY NOTICE > >> This e-mail (including any attachments) is strictly confidential and > >> may also contain privileged information. If you are not the intended > >> recipient you are not authorised to read, print, save, process or > disclose > >> this message. If you have received this message by mistake, please > inform > >> the sender immediately and destroy this e-mail, its attachments and any > >> copies. Any use, distribution, reproduction or disclosure by any person > >> other than the intended recipient is strictly prohibited and the person > >> responsible may incur in penalties. > >> The use of this e-mail is only for professional purposes; there is > no > >> guarantee that the correspondence towards this e-mail will be read only > by > >> the recipient, because, under certain circumstances, there may be a > need to > >> access this email by third subjects belonging to the Company. > >> > >> > >> ________________________________ > >> > >> CONFIDENTIALITY & PRIVACY NOTICE > >> This e-mail (including any attachments) is strictly confidential and may > >> also contain privileged information. If you are not the intended > recipient > >> you are not authorised to read, print, save, process or disclose this > >> message. If you have received this message by mistake, please inform the > >> sender immediately and destroy this e-mail, its attachments and any > copies. > >> Any use, distribution, reproduction or disclosure by any person other > than > >> the intended recipient is strictly prohibited and the person responsible > >> may incur in penalties. > >> The use of this e-mail is only for professional purposes; there is no > >> guarantee that the correspondence towards this e-mail will be read only > by > >> the recipient, because, under certain circumstances, there may be a > need to > >> access this email by third subjects belonging to the Company. > >> > > >
