Re: Enabling Auth between Zookeeper Servers
Hello, I think I found the issue... One can't use the same username for clients and quorums. I configured all of them to be "zookeeper", but in the server-part of the jaas.conf it should probably be more like "kafka" as it's Kafka which authenticates to the zookeeper in that case and zookeepers are using the qorum-part to authenticate to each other. Correct? If that's correct the exception message is completely wrong. It can find the file, it can read the file and it even finds the server-part, but the server-part itself has wrong configuration. At least with the hanged username in the server-part I got a new exception: 2020-02-17 19:28:17,994 [myid:1] - ERROR [main:ZooKeeperServerMain@83] - Unexpected exception, exiting abnormally java.io.IOException: No snapshot found, but there are log entries. Something is broken! Which was probably caused by non-cleaned folders of some previous deployments. So I added the "snapshot.trust.empty=true" to the config to have it start and rebuild the snapshot. And now my zookeeper is running just fine! :) @Mate: as I copied the jaas.conf from your repo is that the exact file you used for testing? Because changing the "user_zookeeper" to "user_kafka" in the server-part fixed it. My next task now is to get Kafka authenticated to zookeeper and get ACLs working. Will be fun :) And I should probably create a ticket to get the jaas.conf-error message fixed!? Best regards Sebastian On 17-Feb-20 1:50 PM, Sebastian Schmitz wrote: Hey, I also just tried using 3.5.7, but same problem... Best regards Sebastian On 17-Feb-20 11:34 AM, Sebastian Schmitz wrote: Hi Mate, that's what I also tried. I copied it to the /opt/zookeeper-cluster/-folder and got the same exception just with the new path. So, if that config works on your side it might be my environment then!? Maybe it's a problem with the base-image openjdk:11-jre-stretch which I use for the container... I'll try using the openjdk:8u222-jre you're using. Best regards Sebastian On 17-Feb-20 9:19 AM, Szalay-Bekő Máté wrote: Hi Sebastian, It's strange indeed... I also see the owner is root. That should work in docker usually, given that you run the zookeeper process with the root user. Maybe copying it to a different folder? I see that the conf folder has different owner, maybe the java security library doesn't like that? But honestly, I don't have any useful explanation. Good luck! Mate On Sun, Feb 16, 2020, 20:06 Sebastian Schmitz < sebastian.schm...@propellerhead.co.nz> wrote: Hey Mate, now it gets really weird. I get the file not found exception: '.20-02-16 18:27:50,530 [myid:1] - ERROR [main:ServerCnxnFactory@246] - No JAAS configuration section named 'Server' was found in '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf java.lang.SecurityException: java.io.IOException: /opt/zookeeper-cluster/zookeeper/conf/jaas.conf (No such file or directory) at java.base/sun.security.provider.ConfigFile$Spi.(Unknown Source) at java.base/sun.security.provider.ConfigFile.(Unknown Source) at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) at java.base/java.lang.reflect.Constructor.newInstance(Unknown Source) at java.base/java.lang.Class.newInstance(Unknown Source) at java.base/javax.security.auth.login.Configuration$2.run(Unknown Source) at java.base/javax.security.auth.login.Configuration$2.run(Unknown Source) at java.base/java.security.AccessController.doPrivileged(Native Method) at java.base/javax.security.auth.login.Configuration.getConfiguration(Unknown Source) at org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:210) at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:646) at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:143) at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:106) at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:64) at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:128) at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82) Caused by: java.io.IOException: /opt/zookeeper-cluster/zookeeper/conf/jaas.conf (No such file or directory) at java.base/sun.security.provider.ConfigFile$Spi.ioException(Unknown Source) at java.base/sun.security.provider.ConfigFile$Spi.init(Unknown Sourc
Re: Enabling Auth between Zookeeper Servers
Hey, I also just tried using 3.5.7, but same problem... Best regards Sebastian On 17-Feb-20 11:34 AM, Sebastian Schmitz wrote: Hi Mate, that's what I also tried. I copied it to the /opt/zookeeper-cluster/-folder and got the same exception just with the new path. So, if that config works on your side it might be my environment then!? Maybe it's a problem with the base-image openjdk:11-jre-stretch which I use for the container... I'll try using the openjdk:8u222-jre you're using. Best regards Sebastian On 17-Feb-20 9:19 AM, Szalay-Bekő Máté wrote: Hi Sebastian, It's strange indeed... I also see the owner is root. That should work in docker usually, given that you run the zookeeper process with the root user. Maybe copying it to a different folder? I see that the conf folder has different owner, maybe the java security library doesn't like that? But honestly, I don't have any useful explanation. Good luck! Mate On Sun, Feb 16, 2020, 20:06 Sebastian Schmitz < sebastian.schm...@propellerhead.co.nz> wrote: Hey Mate, now it gets really weird. I get the file not found exception: '.20-02-16 18:27:50,530 [myid:1] - ERROR [main:ServerCnxnFactory@246] - No JAAS configuration section named 'Server' was found in '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf java.lang.SecurityException: java.io.IOException: /opt/zookeeper-cluster/zookeeper/conf/jaas.conf (No such file or directory) at java.base/sun.security.provider.ConfigFile$Spi.(Unknown Source) at java.base/sun.security.provider.ConfigFile.(Unknown Source) at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) at java.base/java.lang.reflect.Constructor.newInstance(Unknown Source) at java.base/java.lang.Class.newInstance(Unknown Source) at java.base/javax.security.auth.login.Configuration$2.run(Unknown Source) at java.base/javax.security.auth.login.Configuration$2.run(Unknown Source) at java.base/java.security.AccessController.doPrivileged(Native Method) at java.base/javax.security.auth.login.Configuration.getConfiguration(Unknown Source) at org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:210) at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:646) at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:143) at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:106) at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:64) at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:128) at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82) Caused by: java.io.IOException: /opt/zookeeper-cluster/zookeeper/conf/jaas.conf (No such file or directory) at java.base/sun.security.provider.ConfigFile$Spi.ioException(Unknown Source) at java.base/sun.security.provider.ConfigFile$Spi.init(Unknown Source) ... 18 more 2020-02-16 18:27:50,566 [myid:1] - ERROR [main:ZooKeeperServerMain@83] - Unexpected exception, exiting abnormally java.io.IOException: No JAAS configuration section named 'Server' was found in '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf '. at org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:247) at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:646) at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:143) at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:106) at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:64) at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:128) at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82) So I checked the image: root@2eeeb625500c:/opt/zookeeper-cluster/zookeeper/conf# ls -la total 32 drwxr-xr-x 2 1010 1011 4096 Feb 16 18:27 . drwxr-xr-x 12 root root 4096 Feb 16 18:27 .. -rw-r--r-- 1 1010 1011 535 Jan 30 12:18 configuration.xsl -rw-r--r-- 1 root root 600 Feb 13 18:32 jaas.conf -rw-r--r-- 1 root root 101 Feb 11 00:05 java.env -rw-r--r-- 1 1010 1011 2712 Feb 14 05:49 log4j.properties -rw-r--r-- 1 root root 1255 Feb 16 18:27 zoo.cfg -rw-r--r-- 1 1010 1011 922 Feb 14 05:49 zoo_sample.cfg And tried to output the file it
Re: Enabling Auth between Zookeeper Servers
Hi Mate, that's what I also tried. I copied it to the /opt/zookeeper-cluster/-folder and got the same exception just with the new path. So, if that config works on your side it might be my environment then!? Maybe it's a problem with the base-image openjdk:11-jre-stretch which I use for the container... I'll try using the openjdk:8u222-jre you're using. Best regards Sebastian On 17-Feb-20 9:19 AM, Szalay-Bekő Máté wrote: Hi Sebastian, It's strange indeed... I also see the owner is root. That should work in docker usually, given that you run the zookeeper process with the root user. Maybe copying it to a different folder? I see that the conf folder has different owner, maybe the java security library doesn't like that? But honestly, I don't have any useful explanation. Good luck! Mate On Sun, Feb 16, 2020, 20:06 Sebastian Schmitz < sebastian.schm...@propellerhead.co.nz> wrote: Hey Mate, now it gets really weird. I get the file not found exception: '.20-02-16 18:27:50,530 [myid:1] - ERROR [main:ServerCnxnFactory@246] - No JAAS configuration section named 'Server' was found in '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf java.lang.SecurityException: java.io.IOException: /opt/zookeeper-cluster/zookeeper/conf/jaas.conf (No such file or directory) at java.base/sun.security.provider.ConfigFile$Spi.(Unknown Source) at java.base/sun.security.provider.ConfigFile.(Unknown Source) at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) at java.base/java.lang.reflect.Constructor.newInstance(Unknown Source) at java.base/java.lang.Class.newInstance(Unknown Source) at java.base/javax.security.auth.login.Configuration$2.run(Unknown Source) at java.base/javax.security.auth.login.Configuration$2.run(Unknown Source) at java.base/java.security.AccessController.doPrivileged(Native Method) at java.base/javax.security.auth.login.Configuration.getConfiguration(Unknown Source) at org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:210) at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:646) at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:143) at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:106) at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:64) at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:128) at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82) Caused by: java.io.IOException: /opt/zookeeper-cluster/zookeeper/conf/jaas.conf (No such file or directory) at java.base/sun.security.provider.ConfigFile$Spi.ioException(Unknown Source) at java.base/sun.security.provider.ConfigFile$Spi.init(Unknown Source) ... 18 more 2020-02-16 18:27:50,566 [myid:1] - ERROR [main:ZooKeeperServerMain@83] - Unexpected exception, exiting abnormally java.io.IOException: No JAAS configuration section named 'Server' was found in '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf '. at org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:247) at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:646) at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:143) at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:106) at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:64) at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:128) at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82) So I checked the image: root@2eeeb625500c:/opt/zookeeper-cluster/zookeeper/conf# ls -la total 32 drwxr-xr-x 2 1010 1011 4096 Feb 16 18:27 . drwxr-xr-x 12 root root 4096 Feb 16 18:27 .. -rw-r--r-- 1 1010 1011 535 Jan 30 12:18 configuration.xsl -rw-r--r-- 1 root root 600 Feb 13 18:32 jaas.conf -rw-r--r-- 1 root root 101 Feb 11 00:05 java.env -rw-r--r-- 1 1010 1011 2712 Feb 14 05:49 log4j.properties -rw-r--r-- 1 root root 1255 Feb 16 18:27 zoo.cfg -rw-r--r-- 1 1010 1011 922 Feb 14 05:49 zoo_sample.cfg And tried to output the file it states in the error: root@2eeeb625500c:/opt/zookeeper-cluster/zookeeper/conf# cat /opt/zookeeper-cluster/zookeeper/conf/jaas.conf QuorumServer { org.apache.zookeeper.server.auth.DigestLogin
Re: Enabling Auth between Zookeeper Servers
kerberos in production. If it works, I will share my configs / dockerfiles and send you a patched version with more debug info printed out. Why would configuring quorum-auth also enable client-server-auth? it is not very logical indeed... if I see it right, based on the code once you set the java.security.auth.login.config property, then ZooKeeper assumes you want to use server-client sasl authentication. I guess the quorum-auth feature was added later and they introduced 'enable' config property for this, but forget to introduce the same config for the client authentication. I also guess most of the people are interested in the client authentication and it is rare that someone does't need that but needs quorum auth. Still, the current behaviour is not good I think. I will submit a jira ticket requesting an improvement here when I will have time, but feel free to submit it yourself if you wish. Kind regards, Mate On Thu, Feb 13, 2020 at 7:41 PM Sebastian Schmitz < sebastian.schm...@propellerhead.co.nz> wrote: Hey Mate, I checked the java.env-file and it contains: SERVER_JVMFLAGS="-Djava.security.auth.login.config=/opt/zookeeper-cluster/zookeeper/conf/jaas.conf" which is exactly the place where the pasted jaas.conf is placed. I also just changed the config to be saslLoginContext and added the missing semicolon. But I still get the same exception. Why would configuring quorum-auth also enable client-server-auth? Thanks Sebastian On 13-Feb-20 5:50 AM, Szalay-Bekő Máté wrote: Hi Sebastian, thanks for the more details! One thing I found in your config is that you should use: quorum.auth.learner.saslLoginContext=QuorumLearner quorum.auth.server.saslLoginContext=QuorumServer so instead of loginContext, use saslLoginContext in both lines. I found this in the source code, I think the wiki is wrong (I will fix it later). However, actually this didn't really change anything, as the default values are anyway QuorumLearner and QuorumServer, so you can even skip these lines from the config. I think Rakesh is right, you are seeing exceptions related to not the QuorumSasl, but the ClientSasl. This is why ZooKeeper tries to find the 'Server' section (what is configuring the server during the client-server authentication). The name of this section can be overwritten by the "zookeeper.sasl.serverconfig" system property. Based on the exception, ZooKeeper can not find the 'Server' section in the /opt/zookeeper-cluster/zookeeper/conf/jaas.conf file. Are you sure this is the correct jaas.conf? Does the ZooKeeper process have the permissions to open this file? You can specify the jaas config file path for ZooKeeper by providing custom system property e.g. by exporting SERVER_JVMFLAGS="-Djava.security.auth.login.config=/path/to/jaas.conf" before starting zkServer.sh Also in the jaas.conf you copied here, you are missing a semicolon from the end of the last line in the Server block. I am not sure if it is causing any parsing error, but I always add the semicolon to the end of the last line in the block. Mate On Tue, Feb 11, 2020 at 7:53 PM Sebastian Schmitz < sebastian.schm...@propellerhead.co.nz> wrote: Hello Rakesh, as mentioned in the other mail adding the "Server"to jaas.conf didn't help. Here are the Configs and Logs (with the Server-part included): jaas.conf: QuorumServer { org.apache.zookeeper.server.auth.DigestLoginModule required user_zookeeper="test"; }; QuorumClient { org.apache.zookeeper.server.auth.DigestLoginModule required username="zookeeper" password="test"; }; Server { org.apache.zookeeper.server.auth.DigestLoginModule required user_zookeeper="test" }; Client { org.apache.zookeeper.server.auth.DigestLoginModule required username="zookeeper" password="test"; }; zoo.cfg: # The number of milliseconds of each tick tickTime=2000 # The number of ticks that the initial # synchronization phase can take initLimit=10 # The number of ticks that can pass between # sending a request and getting an acknowledgement syncLimit=5 # the directory where the snapshot is stored. # do not use /tmp for storage, /tmp here is just # example sakes. dataDir=/mnt/zk_data # the port at which the clients will connect clientPort=2181 # the maximum number of client connections. # increase this if you need to handle more clients #maxClientCnxns=60 # # Be sure to read the maintenance section of the # administrator guide before turning on autopurge. # # http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance # # The number of snapshots to retain in dataDir #autopurge.snapRetainCount=3 # Purge task interval in hours # Set to "0" to disable auto purge feature #autopurge.purgeInterval=1 dataLogDir=/mnt/zk_data_log autopurge.snapRetai
Re: Enabling Auth between Zookeeper Servers
Hey Mate, I checked the java.env-file and it contains: SERVER_JVMFLAGS="-Djava.security.auth.login.config=/opt/zookeeper-cluster/zookeeper/conf/jaas.conf" which is exactly the place where the pasted jaas.conf is placed. I also just changed the config to be saslLoginContext and added the missing semicolon. But I still get the same exception. Why would configuring quorum-auth also enable client-server-auth? Thanks Sebastian On 13-Feb-20 5:50 AM, Szalay-Bekő Máté wrote: Hi Sebastian, thanks for the more details! One thing I found in your config is that you should use: quorum.auth.learner.saslLoginContext=QuorumLearner quorum.auth.server.saslLoginContext=QuorumServer so instead of loginContext, use saslLoginContext in both lines. I found this in the source code, I think the wiki is wrong (I will fix it later). However, actually this didn't really change anything, as the default values are anyway QuorumLearner and QuorumServer, so you can even skip these lines from the config. I think Rakesh is right, you are seeing exceptions related to not the QuorumSasl, but the ClientSasl. This is why ZooKeeper tries to find the 'Server' section (what is configuring the server during the client-server authentication). The name of this section can be overwritten by the "zookeeper.sasl.serverconfig" system property. Based on the exception, ZooKeeper can not find the 'Server' section in the /opt/zookeeper-cluster/zookeeper/conf/jaas.conf file. Are you sure this is the correct jaas.conf? Does the ZooKeeper process have the permissions to open this file? You can specify the jaas config file path for ZooKeeper by providing custom system property e.g. by exporting SERVER_JVMFLAGS="-Djava.security.auth.login.config=/path/to/jaas.conf" before starting zkServer.sh Also in the jaas.conf you copied here, you are missing a semicolon from the end of the last line in the Server block. I am not sure if it is causing any parsing error, but I always add the semicolon to the end of the last line in the block. Mate On Tue, Feb 11, 2020 at 7:53 PM Sebastian Schmitz < sebastian.schm...@propellerhead.co.nz> wrote: Hello Rakesh, as mentioned in the other mail adding the "Server"to jaas.conf didn't help. Here are the Configs and Logs (with the Server-part included): jaas.conf: QuorumServer { org.apache.zookeeper.server.auth.DigestLoginModule required user_zookeeper="test"; }; QuorumClient { org.apache.zookeeper.server.auth.DigestLoginModule required username="zookeeper" password="test"; }; Server { org.apache.zookeeper.server.auth.DigestLoginModule required user_zookeeper="test" }; Client { org.apache.zookeeper.server.auth.DigestLoginModule required username="zookeeper" password="test"; }; zoo.cfg: # The number of milliseconds of each tick tickTime=2000 # The number of ticks that the initial # synchronization phase can take initLimit=10 # The number of ticks that can pass between # sending a request and getting an acknowledgement syncLimit=5 # the directory where the snapshot is stored. # do not use /tmp for storage, /tmp here is just # example sakes. dataDir=/mnt/zk_data # the port at which the clients will connect clientPort=2181 # the maximum number of client connections. # increase this if you need to handle more clients #maxClientCnxns=60 # # Be sure to read the maintenance section of the # administrator guide before turning on autopurge. # # http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance # # The number of snapshots to retain in dataDir #autopurge.snapRetainCount=3 # Purge task interval in hours # Set to "0" to disable auto purge feature #autopurge.purgeInterval=1 dataLogDir=/mnt/zk_data_log autopurge.snapRetainCount=3 autopurge.purgeInterval=24 quorum.auth.enableSasl=true quorum.auth.learnerRequireSasl=false quorum.auth.serverRequireSasl=false quorum.auth.learner.loginContext=QuorumLearner quorum.auth.server.loginContext=QuorumServer quorum.cnxn.threads.size=20 authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider secureClientPort=2281 server.1=0.0.0.0:2888:3888 server.2=kafkad02.x.azure.com:2888:3888 server.3=kafkad03.x.azure.com:2888:3888 Server-Log: Using config: /opt/zookeeper-cluster/zookeeper/bin/../conf/zoo.cfg Feb 11, 2020 18:43:53 + [1 1] com.newrelic INFO: New Relic Agent: Loading configuration file "/opt/zookeeper-cluster/newrelic/./newrelic.yml" Feb 11, 2020 18:43:53 + [1 1] com.newrelic INFO: Using default collector host: collector.newrelic.com Feb 11, 2020 18:43:53 + [1 1] com.newrelic INFO: New Relic Agent: Writing to log file: /opt/zookeeper-cluster/newrelic/logs/newrelic_agent.log WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by com.newrelic.weave.weavepackage.NewCl
Re: Enabling Auth between Zookeeper Servers
ation section named 'Server' was foundin '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf '. at org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:211) at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:82) at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:133) at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:114) at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:81) Best regards Sebastian On 12-Feb-20 2:36 AM, Rakesh Radhakrishnan wrote: >>>>>java.io.IOException: No JAAS configuration section named 'Server' I could see you have enabled client-server authentication as well. It looks to me that the error is coming from that. Please share the complete error logs to trace it. Have you configured "*Server*" section along with the "*QuorumServer*" and "*QuorumClient*" sections? If not, please configure "*Server*" section along with others and try it out. Reference: https://cwiki.apache.org/confluence/display/ZOOKEEPER/Client-Server+mutual+authentication image.png Thanks, Rakesh On Tue, Feb 11, 2020 at 7:26 AM Sebastian Schmitz <mailto:sebastian.schm...@propellerhead.co.nz>> wrote: Hello, I'm currently looking into enabling the Auth between Zookeeper-Servers and found this documentation: https://cwiki.apache.org/confluence/display/ZOOKEEPER/Server-Server+mutual+authentication However, when I use the config from the document (for Digest-MD5) I get this exception in Zookeeper 3.4.14 and also 3.5.6, which I tried because I thought using latest version could help: java.io.IOException: No JAAS configuration section named 'Server' was found in '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf And of course that's right, because there's only QuorumServer and QuorumClient in the jaas.conf: jaas.conf: QuorumServer { org.apache.zookeeper.server.auth.DigestLoginModule required user_zookeeper="test"; }; QuorumClient { org.apache.zookeeper.server.auth.DigestLoginModule required username="zookeeper" password="test"; }; I also tried renaming the QuorumServer to just "Server". No change. My zoo.cfg: tickTime=2000 initLimit=10 syncLimit=5 dataDir=/mnt/zk_data clientPort=2181 dataLogDir=/mnt/zk_data_log autopurge.snapRetainCount=3 autopurge.purgeInterval=24 quorum.auth.enableSasl=true quorum.auth.learnerRequireSasl=false quorum.auth.serverRequireSasl=false quorum.auth.learner.loginContext=QuorumLearner quorum.auth.server.loginContext=QuorumServer quorum.cnxn.threads.size=20 authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider secureClientPort=2281 server.1=0.0.0.0:2888:3888 Any idea what I could try? Or maybe there's some better document on how to achieve this? Thank you Sebastian -- DISCLAIMER This email contains information that is confidential and which may be legally privileged. If you have received this email in error please notify the sender immediately and delete the email. This email is intended solely for the use of the intended recipient and you may not use or disclose this email in any way. -- DISCLAIMER This email contains information that is confidential and which may be legally privileged. If you have received this email in error please notify the sender immediately and delete the email. This email is intended solely for the use of the intended recipient and you may not use or disclose this email in any way.
Re: Enabling Auth between Zookeeper Servers
59,749 [myid:1] - INFO [main:QuorumPeerMain@130] - Starting quorum peer 2020-02-11 18:43:59,788 [myid:1] - INFO [main:ServerCnxnFactory@117] - Using org.apache.zookeeper.server.NIOServerCnxnFactory as server connection factory 2020-02-11 18:43:59,804 [myid:1] - INFO [PurgeTask:DatadirCleanupManager$PurgeTask@144] - Purge task completed. '.20-02-11 18:43:59,826 [myid:1] - ERROR [main:ServerCnxnFactory@210] - No JAAS configuration section named 'Server' was foundin '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf 2020-02-11 18:43:59,827 [myid:1] - ERROR [main:QuorumPeerMain@92] - Unexpected exception, exiting abnormally java.io.IOException: No JAAS configuration section named 'Server' was foundin '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf '. at org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:211) at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:82) at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:133) at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:114) at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:81) Best regards Sebastian On 11-Feb-20 9:56 PM, Szalay-Bekő Máté wrote: Hello Sebastian, In general I think you did configure ZooKeeper just fine. A few remarks: - I am not sure how ZooKeeper server-server authentication is expected to work when you only use a single server. Would you mind trying to start a Quorum with e.g. 3 servers? - also, I think it is a good idea to avoid using 0.0.0.0 as hostname, especially if you would run the cluster on multiple different servers / docker containers. Try using the fully qualified domain name for a multi-server setup, or if you just test multiple ZooKeeper servers on the same machine, then just use 127.0.0.1. (maybe it has no effect in the current case, but for SSL or for dynamic reconfig it might be bad to use 0.0.0.0. Also I remember problems with rolling restarts when using 0.0.0.0 in the config) - is there a reason why you set 'quorum.auth.learnerRequireSasl' and 'quorum.auth.serverRequireSasl' to false? Using false is usually good idea during rolling upgrade, but if you start a new cluster and want to use server-server authentication, then you can just set them to true. I don't understand why you got the exception " No JAAS configuration section named 'Server' was found" ... Setting the loginContext should have fixed that. If you still see the same issue with the 3 server setup, then can you please share the config files, the command how you start ZooKeeper and also the log files with us, so that we can look deeper? Kind regards, Mate On Tue, Feb 11, 2020 at 2:56 AM Sebastian Schmitz < sebastian.schm...@propellerhead.co.nz> wrote: Hello, I'm currently looking into enabling the Auth between Zookeeper-Servers and found this documentation: https://cwiki.apache.org/confluence/display/ZOOKEEPER/Server-Server+mutual+authentication However, when I use the config from the document (for Digest-MD5) I get this exception in Zookeeper 3.4.14 and also 3.5.6, which I tried because I thought using latest version could help: java.io.IOException: No JAAS configuration section named 'Server' was found in '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf And of course that's right, because there's only QuorumServer and QuorumClient in the jaas.conf: jaas.conf: QuorumServer { org.apache.zookeeper.server.auth.DigestLoginModule required user_zookeeper="test"; }; QuorumClient { org.apache.zookeeper.server.auth.DigestLoginModule required username="zookeeper" password="test"; }; I also tried renaming the QuorumServer to just "Server". No change. My zoo.cfg: tickTime=2000 initLimit=10 syncLimit=5 dataDir=/mnt/zk_data clientPort=2181 dataLogDir=/mnt/zk_data_log autopurge.snapRetainCount=3 autopurge.purgeInterval=24 quorum.auth.enableSasl=true quorum.auth.learnerRequireSasl=false quorum.auth.serverRequireSasl=false quorum.auth.learner.loginContext=QuorumLearner quorum.auth.server.loginContext=QuorumServer quorum.cnxn.threads.size=20 authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider secureClientPort=2281 server.1=0.0.0.0:2888:3888 Any idea what I could try? Or maybe there's some better document on how to achieve this? Thank you Sebastian -- DISCLAIMER This email contains information that is confidential and which may be legally privileged. If you have received this email in error please notify the sender immediately and delete the email. This email is intended solely for the use of the intended recipient and you may not use or disclose this email in any way. -- DISCLAIMER This email contains information that is confidential and which may be legally privileged. If you have received this email in er
Enabling Auth between Zookeeper Servers
Hello, I'm currently looking into enabling the Auth between Zookeeper-Servers and found this documentation: https://cwiki.apache.org/confluence/display/ZOOKEEPER/Server-Server+mutual+authentication However, when I use the config from the document (for Digest-MD5) I get this exception in Zookeeper 3.4.14 and also 3.5.6, which I tried because I thought using latest version could help: java.io.IOException: No JAAS configuration section named 'Server' was found in '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf And of course that's right, because there's only QuorumServer and QuorumClient in the jaas.conf: jaas.conf: QuorumServer { org.apache.zookeeper.server.auth.DigestLoginModule required user_zookeeper="test"; }; QuorumClient { org.apache.zookeeper.server.auth.DigestLoginModule required username="zookeeper" password="test"; }; I also tried renaming the QuorumServer to just "Server". No change. My zoo.cfg: tickTime=2000 initLimit=10 syncLimit=5 dataDir=/mnt/zk_data clientPort=2181 dataLogDir=/mnt/zk_data_log autopurge.snapRetainCount=3 autopurge.purgeInterval=24 quorum.auth.enableSasl=true quorum.auth.learnerRequireSasl=false quorum.auth.serverRequireSasl=false quorum.auth.learner.loginContext=QuorumLearner quorum.auth.server.loginContext=QuorumServer quorum.cnxn.threads.size=20 authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider secureClientPort=2281 server.1=0.0.0.0:2888:3888 Any idea what I could try? Or maybe there's some better document on how to achieve this? Thank you Sebastian -- DISCLAIMER This email contains information that is confidential and which may be legally privileged. If you have received this email in error please notify the sender immediately and delete the email. This email is intended solely for the use of the intended recipient and you may not use or disclose this email in any way.
Re: Zookeeper 3.5.6 - Exception causing close of session 0x0: null
Hello Enrico, yes, a custom made client, that was updated and now is stopped again... The difference is that in new version only this is the output: Oct 25 10:56:23 zookeeper_node_1: 2019-10-24 21:56:23,119 [myid:1] - WARN [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of session 0x0: null In the 3.4.14 it was also stating where the connection came from before giving the warning: Oct 21 16:02:16 zookeeper_node_2: 2019-10-2103:02:16,679 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@222] - Accepted socket connection from /10.24.8.13:60517 Best regards Sebastian On 29-Oct-19 7:06 PM, Enrico Olivelli wrote: Il mar 29 ott 2019, 00:52 Sebastian Schmitz < sebastian.schm...@propellerhead.co.nz> ha scritto: Hello again, just as a final update on this. We had a weird consumer which someone updated without informing me... Can you please define 'weird'? A custom made client? Turning it off also turned off the errors from spamming the logs. However it seems like the 3.5.6 is less informative than the 3.4.14 we had before as 3.4.14 also had the IP-Address when the connection was opened in the messages and not just the warnings... If you can point to the differences we can bring back precious logs Great to see you found the problem Enrico Best regards Sebastian On 25-Oct-19 1:26 PM, Sebastian Schmitz wrote: Hello Enrico, thanks for the reply. I'll try to find the exception, there was none around 14:50 when it first appeared. I also try to find the client that's connecting just before that as it's not a Kafka-node: Oct 21 14:50:31 zookeeper_node_3: 2019-10-21 01:50:31,532 [myid:3] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@222] - Accepted socket connection from /10.24.8.13:64261 Oct 21 14:50:31 zookeeper_node_3: 2019-10-21 01:50:31,544 [myid:3] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@383] - Exception causing close of session 0x0: null Oct 21 14:50:31 zookeeper_node_3: 2019-10-21 01:50:31,545 [myid:3] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1056] - Closed socket connection for client /10.24.8.13:64261 (no session established for client) The same IP is able to connect to Zookeeper successfully some time before: Oct 21 13:18:44 zookeeper_node_3: 2019-10-21 00:18:44,823 [myid:3] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@222] - Accepted socket connection from /10.24.8.13:50212 Oct 21 13:18:44 zookeeper_node_3: 2019-10-21 00:18:44,825 [myid:3] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@949] - Client attempting to establish new session at /10.24.8.13:50212 Oct 21 13:18:44 zookeeper_node_3: 2019-10-21 00:18:44,833 [myid:3] - INFO [CommitProcessor:3:ZooKeeperServer@694] - Established session 0x30b2dfdd8ac with negotiated timeout 1 for client /10.24.8.13:50212 Oct 21 13:18:53 zookeeper_node_3: 2019-10-21 00:18:53,374 [myid:3] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@222] - Accepted socket connection from /10.24.8.13:50314 Oct 21 13:18:53 zookeeper_node_3: 2019-10-21 00:18:53,374 [myid:3] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@949] - Client attempting to establish new session at /10.24.8.13:50314 Oct 21 13:18:53 zookeeper_node_3: 2019-10-21 00:18:53,385 [myid:3] - INFO [CommitProcessor:3:ZooKeeperServer@694] - Established session 0x30b2dfdd8ac0001 with negotiated timeout 1 for client /10.24.8.13:50314 Oct 21 13:18:53 zookeeper_node_3: 2019-10-21 00:18:53,588 [myid:3] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1056] - Closed socket connection for client /10.24.8.13:50314 which had sessionid 0x30b2dfdd8ac0001 I also checked the Logs from the update I did on Wednesday night to update to 3.5.6. This is the Log of the start of zookeeper_node_1 until the errors show up... Can't see any exceptions :( Oct 23 02:07:03 zookeeper_node_1: setting ZOO_LOG4J_PROP=INFO,CONSOLE Oct 23 02:07:03 zookeeper_node_1: ZooKeeper JMX enabled by default Oct 23 02:07:03 zookeeper_node_1: Using config: /opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../conf/zoo.cfg Oct 23 02:07:05 zookeeper_node_1: Oct 22, 2019 13:07:05 + [1 1] com.newrelic INFO: New Relic Agent: Loading configuration file "/opt/zookeeper-cluster/newrelic/./newrelic.yml" Oct 23 02:07:05 zookeeper_node_1: Oct 22, 2019 13:07:05 + [1 1] com.newrelic INFO: Using default collector host: collector.newrelic.com Oct 23 02:07:05 zookeeper_node_1: Oct 22, 2019 13:07:05 + [1 1] com.newrelic INFO: New Relic Agent: Writing to log file: /opt/zookeeper-cluster/newrelic/logs/newrelic_agent.log Oct 23 02:07:28 zookeeper_node_1: 2019-10-22 13:07:28,743 [myid:] - INFO [main:QuorumPeerConfig@133] - Reading configuration from: /opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../conf/zoo.cfg Oct 23 02:07:28 zookeeper_node_1: 2019-10-22 13:07:28,76
Re: Zookeeper 3.5.6 - Exception causing close of session 0x0: null
Hello again, just as a final update on this. We had a weird consumer which someone updated without informing me... Turning it off also turned off the errors from spamming the logs. However it seems like the 3.5.6 is less informative than the 3.4.14 we had before as 3.4.14 also had the IP-Address when the connection was opened in the messages and not just the warnings... Best regards Sebastian On 25-Oct-19 1:26 PM, Sebastian Schmitz wrote: Hello Enrico, thanks for the reply. I'll try to find the exception, there was none around 14:50 when it first appeared. I also try to find the client that's connecting just before that as it's not a Kafka-node: Oct 21 14:50:31 zookeeper_node_3: 2019-10-21 01:50:31,532 [myid:3] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@222] - Accepted socket connection from /10.24.8.13:64261 Oct 21 14:50:31 zookeeper_node_3: 2019-10-21 01:50:31,544 [myid:3] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@383] - Exception causing close of session 0x0: null Oct 21 14:50:31 zookeeper_node_3: 2019-10-21 01:50:31,545 [myid:3] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1056] - Closed socket connection for client /10.24.8.13:64261 (no session established for client) The same IP is able to connect to Zookeeper successfully some time before: Oct 21 13:18:44 zookeeper_node_3: 2019-10-21 00:18:44,823 [myid:3] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@222] - Accepted socket connection from /10.24.8.13:50212 Oct 21 13:18:44 zookeeper_node_3: 2019-10-21 00:18:44,825 [myid:3] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@949] - Client attempting to establish new session at /10.24.8.13:50212 Oct 21 13:18:44 zookeeper_node_3: 2019-10-21 00:18:44,833 [myid:3] - INFO [CommitProcessor:3:ZooKeeperServer@694] - Established session 0x30b2dfdd8ac with negotiated timeout 1 for client /10.24.8.13:50212 Oct 21 13:18:53 zookeeper_node_3: 2019-10-21 00:18:53,374 [myid:3] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@222] - Accepted socket connection from /10.24.8.13:50314 Oct 21 13:18:53 zookeeper_node_3: 2019-10-21 00:18:53,374 [myid:3] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@949] - Client attempting to establish new session at /10.24.8.13:50314 Oct 21 13:18:53 zookeeper_node_3: 2019-10-21 00:18:53,385 [myid:3] - INFO [CommitProcessor:3:ZooKeeperServer@694] - Established session 0x30b2dfdd8ac0001 with negotiated timeout 1 for client /10.24.8.13:50314 Oct 21 13:18:53 zookeeper_node_3: 2019-10-21 00:18:53,588 [myid:3] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1056] - Closed socket connection for client /10.24.8.13:50314 which had sessionid 0x30b2dfdd8ac0001 I also checked the Logs from the update I did on Wednesday night to update to 3.5.6. This is the Log of the start of zookeeper_node_1 until the errors show up... Can't see any exceptions :( Oct 23 02:07:03 zookeeper_node_1: setting ZOO_LOG4J_PROP=INFO,CONSOLE Oct 23 02:07:03 zookeeper_node_1: ZooKeeper JMX enabled by default Oct 23 02:07:03 zookeeper_node_1: Using config: /opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../conf/zoo.cfg Oct 23 02:07:05 zookeeper_node_1: Oct 22, 2019 13:07:05 + [1 1] com.newrelic INFO: New Relic Agent: Loading configuration file "/opt/zookeeper-cluster/newrelic/./newrelic.yml" Oct 23 02:07:05 zookeeper_node_1: Oct 22, 2019 13:07:05 + [1 1] com.newrelic INFO: Using default collector host: collector.newrelic.com Oct 23 02:07:05 zookeeper_node_1: Oct 22, 2019 13:07:05 + [1 1] com.newrelic INFO: New Relic Agent: Writing to log file: /opt/zookeeper-cluster/newrelic/logs/newrelic_agent.log Oct 23 02:07:28 zookeeper_node_1: 2019-10-22 13:07:28,743 [myid:] - INFO [main:QuorumPeerConfig@133] - Reading configuration from: /opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../conf/zoo.cfg Oct 23 02:07:28 zookeeper_node_1: 2019-10-22 13:07:28,764 [myid:] - INFO [main:QuorumPeerConfig@385] - clientPortAddress is 0.0.0.0/0.0.0.0:2181 Oct 23 02:07:28 zookeeper_node_1: 2019-10-22 13:07:28,764 [myid:] - INFO [main:QuorumPeerConfig@389] - secureClientPort is not set Oct 23 02:07:28 zookeeper_node_1: 2019-10-22 13:07:28,877 [myid:1] - INFO [main:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3 Oct 23 02:07:28 zookeeper_node_1: 2019-10-22 13:07:28,878 [myid:1] - INFO [main:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0 Oct 23 02:07:28 zookeeper_node_1: 2019-10-22 13:07:28,878 [myid:1] - INFO [main:DatadirCleanupManager@101] - Purge task is not scheduled. Oct 23 02:07:28 zookeeper_node_1: 2019-10-22 13:07:28,889 [myid:1] - INFO [main:ManagedUtil@46] - Log4j found with jmx enabled. Oct 23 02:07:29 zookeeper_node_1: 2019-10-22 13:07:29,046 [myid:1] - INFO [main:QuorumPeerMain@141] - Starting quorum peer Oct 2
Re: Zookeeper 3.5.6 - Exception causing close of session 0x0: null
f session 0x0: null Oct 23 02:07:46 zookeeper_node_1: 2019-10-22 13:07:46,474 [myid:1] - WARN [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 23 02:07:47 zookeeper_node_1: 2019-10-22 13:07:47,111 [myid:1] - WARN [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 23 02:07:47 zookeeper_node_1: 2019-10-22 13:07:47,292 [myid:1] - WARN [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of session 0x0: null Thanks Sebastian On 25-Oct-19 11:17 AM, Enrico Olivelli wrote: Sebastian This 'null' looks like a NullPointerException (this kind f exceptions tracks only the full stacktrace at the first occurrence usually) Can you find some full stacktrace of the error? Are you able to enable more detailed logging? Do you see errors on the client (kafka broker?) ? Maybe on oeg.apache.zookeeper.xxx logs Enrico Il ven 25 ott 2019, 00:05 Sebastian Schmitz < sebastian.schm...@propellerhead.co.nz> ha scritto: Hello, I did a rolling upgrade of the Zookeeper from 3.4.14 to 3.5.6 on 23rd in my Test-Environment. It's zookeeping a three node Kafka-Cluster running on 2.3.0. After the update I get spammed with these warnings: Oct 25 10:56:23 zookeeper_node_1: 2019-10-24 21:56:23,119 [myid:1] - WARN [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:23 zookeeper_node_2: 2019-10-24 21:56:23,478 [myid:2] - WARN [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:23 zookeeper_node_3: 2019-10-24 21:56:23,531 [myid:3] - WARN [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:23 zookeeper_node_1: 2019-10-24 21:56:23,776 [myid:1] - WARN [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:23 zookeeper_node_2: 2019-10-24 21:56:23,879 [myid:2] - WARN [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:24 zookeeper_node_3: 2019-10-24 21:56:24,086 [myid:3] - WARN [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:24 zookeeper_node_1: 2019-10-24 21:56:24,090 [myid:1] - WARN [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:24 zookeeper_node_2: 2019-10-24 21:56:24,497 [myid:2] - WARN [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:24 zookeeper_node_3: 2019-10-24 21:56:24,601 [myid:3] - WARN [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:24 zookeeper_node_1: 2019-10-24 21:56:24,798 [myid:1] - WARN [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:24 zookeeper_node_3: 2019-10-24 21:56:24,904 [myid:3] - WARN [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:25 zookeeper_node_2: 2019-10-24 21:56:25,007 [myid:2] - WARN [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:25 zookeeper_node_1: 2019-10-24 21:56:25,260 [myid:1] - WARN [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:25 zookeeper_node_3: 2019-10-24 21:56:25,467 [myid:3] - WARN [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:25 zookeeper_node_2: 2019-10-24 21:56:25,619 [myid:2] - WARN [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:25 zookeeper_node_1: 2019-10-24 21:56:25,666 [myid:1] - WARN [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:26 zookeeper_node_3: 2019-10-24 21:56:25,923 [myid:3] - WARN [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:26 zookeeper_node_2: 2019-10-24 21:56:26,178 [myid:2] - WARN [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:26 zookeeper_node_1: 2019-10-24 21:56:26,329 [myid:1] - WARN [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:26 zookeeper_node_3: 2019-10-24 21:56:26,484 [myid:3] - WARN [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:26 zookeeper_node_2: 2019-10-24 21:56:26,792 [myid:2] - WARN [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:26 zookeeper_node_3: 2019-10-24 21:56:26,890 [myid:3] - WARN [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:26 zookeeper_node_1: 2019-10-24 21:56:26,888 [myid:1] - WARN [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:27 zookeeper_node_2:
Re: Zookeeper 3.5.6 - Exception causing close of session 0x0: null
Hello Again, I have a small correction to make... It didn't start after the upgrade to 3.5.6, but just in the middle of a day at 14:50 on 21st And the upgrade to 3.5.6 didn't automagically solve it So any help to find the cause of this would be great. Thanks Sebastian On 25-Oct-19 11:05 AM, Sebastian Schmitz wrote: Hello, I did a rolling upgrade of the Zookeeper from 3.4.14 to 3.5.6 on 23rd in my Test-Environment. It's zookeeping a three node Kafka-Cluster running on 2.3.0. After the update I get spammed with these warnings: Oct 25 10:56:23 zookeeper_node_1: 2019-10-24 21:56:23,119 [myid:1] - WARN [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:23 zookeeper_node_2: 2019-10-24 21:56:23,478 [myid:2] - WARN [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:23 zookeeper_node_3: 2019-10-24 21:56:23,531 [myid:3] - WARN [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:23 zookeeper_node_1: 2019-10-24 21:56:23,776 [myid:1] - WARN [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:23 zookeeper_node_2: 2019-10-24 21:56:23,879 [myid:2] - WARN [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:24 zookeeper_node_3: 2019-10-24 21:56:24,086 [myid:3] - WARN [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:24 zookeeper_node_1: 2019-10-24 21:56:24,090 [myid:1] - WARN [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:24 zookeeper_node_2: 2019-10-24 21:56:24,497 [myid:2] - WARN [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:24 zookeeper_node_3: 2019-10-24 21:56:24,601 [myid:3] - WARN [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:24 zookeeper_node_1: 2019-10-24 21:56:24,798 [myid:1] - WARN [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:24 zookeeper_node_3: 2019-10-24 21:56:24,904 [myid:3] - WARN [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:25 zookeeper_node_2: 2019-10-24 21:56:25,007 [myid:2] - WARN [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:25 zookeeper_node_1: 2019-10-24 21:56:25,260 [myid:1] - WARN [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:25 zookeeper_node_3: 2019-10-24 21:56:25,467 [myid:3] - WARN [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:25 zookeeper_node_2: 2019-10-24 21:56:25,619 [myid:2] - WARN [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:25 zookeeper_node_1: 2019-10-24 21:56:25,666 [myid:1] - WARN [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:26 zookeeper_node_3: 2019-10-24 21:56:25,923 [myid:3] - WARN [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:26 zookeeper_node_2: 2019-10-24 21:56:26,178 [myid:2] - WARN [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:26 zookeeper_node_1: 2019-10-24 21:56:26,329 [myid:1] - WARN [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:26 zookeeper_node_3: 2019-10-24 21:56:26,484 [myid:3] - WARN [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:26 zookeeper_node_2: 2019-10-24 21:56:26,792 [myid:2] - WARN [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:26 zookeeper_node_3: 2019-10-24 21:56:26,890 [myid:3] - WARN [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:26 zookeeper_node_1: 2019-10-24 21:56:26,888 [myid:1] - WARN [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:27 zookeeper_node_2: 2019-10-24 21:56:27,347 [myid:2] - WARN [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:27 zookeeper_node_1: 2019-10-24 21:56:27,354 [myid:1] - WARN [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:27 zookeeper_node_3: 2019-10-24 21:56:27,662 [myid:3] - WARN [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:27 zookeeper_node_3: 2019-10-24 21:56:27,810 [myid:3] - WARN [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56
Zookeeper 3.5.6 - Exception causing close of session 0x0: null
Hello, I did a rolling upgrade of the Zookeeper from 3.4.14 to 3.5.6 on 23rd in my Test-Environment. It's zookeeping a three node Kafka-Cluster running on 2.3.0. After the update I get spammed with these warnings: Oct 25 10:56:23 zookeeper_node_1: 2019-10-24 21:56:23,119 [myid:1] - WARN [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:23 zookeeper_node_2: 2019-10-24 21:56:23,478 [myid:2] - WARN [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:23 zookeeper_node_3: 2019-10-24 21:56:23,531 [myid:3] - WARN [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:23 zookeeper_node_1: 2019-10-24 21:56:23,776 [myid:1] - WARN [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:23 zookeeper_node_2: 2019-10-24 21:56:23,879 [myid:2] - WARN [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:24 zookeeper_node_3: 2019-10-24 21:56:24,086 [myid:3] - WARN [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:24 zookeeper_node_1: 2019-10-24 21:56:24,090 [myid:1] - WARN [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:24 zookeeper_node_2: 2019-10-24 21:56:24,497 [myid:2] - WARN [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:24 zookeeper_node_3: 2019-10-24 21:56:24,601 [myid:3] - WARN [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:24 zookeeper_node_1: 2019-10-24 21:56:24,798 [myid:1] - WARN [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:24 zookeeper_node_3: 2019-10-24 21:56:24,904 [myid:3] - WARN [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:25 zookeeper_node_2: 2019-10-24 21:56:25,007 [myid:2] - WARN [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:25 zookeeper_node_1: 2019-10-24 21:56:25,260 [myid:1] - WARN [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:25 zookeeper_node_3: 2019-10-24 21:56:25,467 [myid:3] - WARN [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:25 zookeeper_node_2: 2019-10-24 21:56:25,619 [myid:2] - WARN [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:25 zookeeper_node_1: 2019-10-24 21:56:25,666 [myid:1] - WARN [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:26 zookeeper_node_3: 2019-10-24 21:56:25,923 [myid:3] - WARN [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:26 zookeeper_node_2: 2019-10-24 21:56:26,178 [myid:2] - WARN [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:26 zookeeper_node_1: 2019-10-24 21:56:26,329 [myid:1] - WARN [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:26 zookeeper_node_3: 2019-10-24 21:56:26,484 [myid:3] - WARN [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:26 zookeeper_node_2: 2019-10-24 21:56:26,792 [myid:2] - WARN [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:26 zookeeper_node_3: 2019-10-24 21:56:26,890 [myid:3] - WARN [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:26 zookeeper_node_1: 2019-10-24 21:56:26,888 [myid:1] - WARN [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:27 zookeeper_node_2: 2019-10-24 21:56:27,347 [myid:2] - WARN [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:27 zookeeper_node_1: 2019-10-24 21:56:27,354 [myid:1] - WARN [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:27 zookeeper_node_3: 2019-10-24 21:56:27,662 [myid:3] - WARN [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:27 zookeeper_node_3: 2019-10-24 21:56:27,810 [myid:3] - WARN [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:27 zookeeper_node_1: 2019-10-24 21:56:27,811 [myid:1] - WARN [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:27 zookeeper_node_2: 2019-10-24 21:56:27,816 [myid:2] - WARN [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of session 0x0: null Oct 25 10:56:28