Re: Enabling Auth between Zookeeper Servers

2020-02-17 Thread Sebastian Schmitz

Hello,

I think I found the issue...

One can't use the same username for clients and quorums. I configured 
all of them to be "zookeeper", but in the server-part of the jaas.conf 
it should probably be more like "kafka" as it's Kafka which 
authenticates to the zookeeper in that case and zookeepers are using the 
qorum-part to authenticate to each other. Correct?
If that's correct the exception message is completely wrong. It can find 
the file, it can read the file and it even finds the server-part, but 
the server-part itself has wrong configuration.


At least with the hanged username in the server-part I got a new exception:

2020-02-17 19:28:17,994 [myid:1] - ERROR [main:ZooKeeperServerMain@83] - 
Unexpected exception, exiting abnormally
java.io.IOException: No snapshot found, but there are log entries. 
Something is broken!


Which was probably caused by non-cleaned folders of some previous 
deployments. So I added the "snapshot.trust.empty=true" to the config to 
have it start and rebuild the snapshot. And now my zookeeper is running 
just fine! :)


@Mate: as I copied the jaas.conf from your repo is that the exact file 
you used for testing? Because changing the "user_zookeeper" to 
"user_kafka" in the server-part fixed it.


My next task now is to get Kafka authenticated to zookeeper and get ACLs 
working. Will be fun :)
And I should probably create a ticket to get the jaas.conf-error message 
fixed!?


Best regards

Sebastian


On 17-Feb-20 1:50 PM, Sebastian Schmitz wrote:

Hey,

I also just tried using 3.5.7, but same problem...

Best regards

Sebastian


On 17-Feb-20 11:34 AM, Sebastian Schmitz wrote:

Hi Mate,

that's what I also tried. I copied it to the 
/opt/zookeeper-cluster/-folder and got the same exception just with 
the new path.


So, if that config works on your side it might be my environment 
then!? Maybe it's a problem with the base-image 
openjdk:11-jre-stretch which I use for the container... I'll try 
using the openjdk:8u222-jre you're using.


Best regards

Sebastian


On 17-Feb-20 9:19 AM, Szalay-Bekő Máté wrote:

Hi Sebastian,

It's strange indeed... I also see the owner is root. That should 
work in

docker usually, given that you run the zookeeper process with the root
user. Maybe copying it to a different folder? I see that the conf 
folder

has different owner, maybe the java security library doesn't like that?

But honestly, I don't have any useful explanation.

Good luck!
Mate

On Sun, Feb 16, 2020, 20:06 Sebastian Schmitz <
sebastian.schm...@propellerhead.co.nz> wrote:


Hey Mate,

now it gets really weird. I get the file not found exception:

'.20-02-16 18:27:50,530 [myid:1] - ERROR 
[main:ServerCnxnFactory@246] -

No JAAS configuration section named 'Server' was found in
'/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
java.lang.SecurityException: java.io.IOException:
/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
   (No such file or directory)
  at
java.base/sun.security.provider.ConfigFile$Spi.(Unknown Source)
  at java.base/sun.security.provider.ConfigFile.(Unknown
Source)
  at
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native 



Method)
  at
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown 



Source)
  at
java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown 



Source)
  at 
java.base/java.lang.reflect.Constructor.newInstance(Unknown

Source)
  at java.base/java.lang.Class.newInstance(Unknown Source)
  at
java.base/javax.security.auth.login.Configuration$2.run(Unknown 
Source)

  at
java.base/javax.security.auth.login.Configuration$2.run(Unknown 
Source)
  at 
java.base/java.security.AccessController.doPrivileged(Native

Method)
  at
java.base/javax.security.auth.login.Configuration.getConfiguration(Unknown 


Source)
  at

org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:210) 


  at

org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:646) 


  at

org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:143) 


  at

org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:106) 


  at

org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:64) 


  at

org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:128) 


  at

org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82) 


Caused by: java.io.IOException:
/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
   (No such file or directory)
  at
java.base/sun.security.provider.ConfigFile$Spi.ioException(Unknown 
Source)
  at 
java.base/sun.security.provider.ConfigFile$Spi.init(Unknown

Sourc

Re: Enabling Auth between Zookeeper Servers

2020-02-16 Thread Sebastian Schmitz

Hey,

I also just tried using 3.5.7, but same problem...

Best regards

Sebastian


On 17-Feb-20 11:34 AM, Sebastian Schmitz wrote:

Hi Mate,

that's what I also tried. I copied it to the 
/opt/zookeeper-cluster/-folder and got the same exception just with 
the new path.


So, if that config works on your side it might be my environment 
then!? Maybe it's a problem with the base-image openjdk:11-jre-stretch 
which I use for the container... I'll try using the openjdk:8u222-jre 
you're using.


Best regards

Sebastian


On 17-Feb-20 9:19 AM, Szalay-Bekő Máté wrote:

Hi Sebastian,

It's strange indeed... I also see the owner is root. That should work in
docker usually, given that you run the zookeeper process with the root
user. Maybe copying it to a different folder? I see that the conf folder
has different owner, maybe the java security library doesn't like that?

But honestly, I don't have any useful explanation.

Good luck!
Mate

On Sun, Feb 16, 2020, 20:06 Sebastian Schmitz <
sebastian.schm...@propellerhead.co.nz> wrote:


Hey Mate,

now it gets really weird. I get the file not found exception:

'.20-02-16 18:27:50,530 [myid:1] - ERROR [main:ServerCnxnFactory@246] -
No JAAS configuration section named 'Server' was found in
'/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
java.lang.SecurityException: java.io.IOException:
/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
   (No such file or directory)
  at
java.base/sun.security.provider.ConfigFile$Spi.(Unknown Source)
  at java.base/sun.security.provider.ConfigFile.(Unknown
Source)
  at
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native 



Method)
  at
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown 



Source)
  at
java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown 



Source)
  at 
java.base/java.lang.reflect.Constructor.newInstance(Unknown

Source)
  at java.base/java.lang.Class.newInstance(Unknown Source)
  at
java.base/javax.security.auth.login.Configuration$2.run(Unknown Source)
  at
java.base/javax.security.auth.login.Configuration$2.run(Unknown Source)
  at 
java.base/java.security.AccessController.doPrivileged(Native

Method)
  at
java.base/javax.security.auth.login.Configuration.getConfiguration(Unknown 


Source)
  at

org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:210) 


  at

org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:646) 


  at

org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:143) 


  at

org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:106) 


  at

org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:64) 


  at

org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:128) 


  at

org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82) 


Caused by: java.io.IOException:
/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
   (No such file or directory)
  at
java.base/sun.security.provider.ConfigFile$Spi.ioException(Unknown 
Source)
  at 
java.base/sun.security.provider.ConfigFile$Spi.init(Unknown

Source)
  ... 18 more
2020-02-16 18:27:50,566 [myid:1] - ERROR 
[main:ZooKeeperServerMain@83] -

Unexpected exception, exiting abnormally
java.io.IOException: No JAAS configuration section named 'Server' was
found in '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
'.
  at

org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:247) 


  at

org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:646) 


  at

org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:143) 


  at

org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:106) 


  at

org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:64) 


  at

org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:128) 


  at

org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82) 



So I checked the image:

root@2eeeb625500c:/opt/zookeeper-cluster/zookeeper/conf# ls -la
total 32
drwxr-xr-x  2 1010 1011 4096 Feb 16 18:27 .
drwxr-xr-x 12 root root 4096 Feb 16 18:27 ..
-rw-r--r--  1 1010 1011  535 Jan 30 12:18 configuration.xsl
-rw-r--r--  1 root root  600 Feb 13 18:32 jaas.conf
-rw-r--r--  1 root root  101 Feb 11 00:05 java.env
-rw-r--r--  1 1010 1011 2712 Feb 14 05:49 log4j.properties
-rw-r--r--  1 root root 1255 Feb 16 18:27 zoo.cfg
-rw-r--r--  1 1010 1011  922 Feb 14 05:49 zoo_sample.cfg

And tried to output the file it 

Re: Enabling Auth between Zookeeper Servers

2020-02-16 Thread Sebastian Schmitz

Hi Mate,

that's what I also tried. I copied it to the 
/opt/zookeeper-cluster/-folder and got the same exception just with the 
new path.


So, if that config works on your side it might be my environment then!? 
Maybe it's a problem with the base-image openjdk:11-jre-stretch which I 
use for the container... I'll try using the openjdk:8u222-jre you're using.


Best regards

Sebastian


On 17-Feb-20 9:19 AM, Szalay-Bekő Máté wrote:

Hi Sebastian,

It's strange indeed... I also see the owner is root. That should work in
docker usually, given that you run the zookeeper process with the root
user. Maybe copying it to a different folder? I see that the conf folder
has different owner, maybe the java security library doesn't like that?

But honestly, I don't have any useful explanation.

Good luck!
Mate

On Sun, Feb 16, 2020, 20:06 Sebastian Schmitz <
sebastian.schm...@propellerhead.co.nz> wrote:


Hey Mate,

now it gets really weird. I get the file not found exception:

'.20-02-16 18:27:50,530 [myid:1] - ERROR [main:ServerCnxnFactory@246] -
No JAAS configuration section named 'Server' was found in
'/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
java.lang.SecurityException: java.io.IOException:
/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
   (No such file or directory)
  at
java.base/sun.security.provider.ConfigFile$Spi.(Unknown Source)
  at java.base/sun.security.provider.ConfigFile.(Unknown
Source)
  at
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native

Method)
  at
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown

Source)
  at
java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown

Source)
  at java.base/java.lang.reflect.Constructor.newInstance(Unknown
Source)
  at java.base/java.lang.Class.newInstance(Unknown Source)
  at
java.base/javax.security.auth.login.Configuration$2.run(Unknown Source)
  at
java.base/javax.security.auth.login.Configuration$2.run(Unknown Source)
  at java.base/java.security.AccessController.doPrivileged(Native
Method)
  at
java.base/javax.security.auth.login.Configuration.getConfiguration(Unknown
Source)
  at

org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:210)
  at

org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:646)
  at

org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:143)
  at

org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:106)
  at

org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:64)
  at

org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:128)
  at

org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82)
Caused by: java.io.IOException:
/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
   (No such file or directory)
  at
java.base/sun.security.provider.ConfigFile$Spi.ioException(Unknown Source)
  at java.base/sun.security.provider.ConfigFile$Spi.init(Unknown
Source)
  ... 18 more
2020-02-16 18:27:50,566 [myid:1] - ERROR [main:ZooKeeperServerMain@83] -
Unexpected exception, exiting abnormally
java.io.IOException: No JAAS configuration section named 'Server' was
found in '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
'.
  at

org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:247)
  at

org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:646)
  at

org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:143)
  at

org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:106)
  at

org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:64)
  at

org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:128)
  at

org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82)

So I checked the image:

root@2eeeb625500c:/opt/zookeeper-cluster/zookeeper/conf# ls -la
total 32
drwxr-xr-x  2 1010 1011 4096 Feb 16 18:27 .
drwxr-xr-x 12 root root 4096 Feb 16 18:27 ..
-rw-r--r--  1 1010 1011  535 Jan 30 12:18 configuration.xsl
-rw-r--r--  1 root root  600 Feb 13 18:32 jaas.conf
-rw-r--r--  1 root root  101 Feb 11 00:05 java.env
-rw-r--r--  1 1010 1011 2712 Feb 14 05:49 log4j.properties
-rw-r--r--  1 root root 1255 Feb 16 18:27 zoo.cfg
-rw-r--r--  1 1010 1011  922 Feb 14 05:49 zoo_sample.cfg

And tried to output the file it states in the error:

root@2eeeb625500c:/opt/zookeeper-cluster/zookeeper/conf# cat
/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
QuorumServer {
 org.apache.zookeeper.server.auth.DigestLogin

Re: Enabling Auth between Zookeeper Servers

2020-02-16 Thread Sebastian Schmitz
 kerberos
in production. If it works, I will share my configs / dockerfiles and send
you a patched version with more debug info printed out.


Why would configuring quorum-auth also enable client-server-auth?

it is not very logical indeed... if I see it right, based on the code once
you set the java.security.auth.login.config property, then ZooKeeper
assumes you want to use server-client sasl authentication. I guess the
quorum-auth feature was added later and they introduced 'enable' config
property for this, but forget to introduce the same config for the client
authentication. I also guess most of the people are interested in the
client authentication and it is rare that someone does't need that but
needs quorum auth. Still, the current behaviour is not good I think. I will
submit a jira ticket requesting an improvement here when I will have time,
but feel free to submit it yourself if you wish.

Kind regards,
Mate

On Thu, Feb 13, 2020 at 7:41 PM Sebastian Schmitz <
sebastian.schm...@propellerhead.co.nz> wrote:


Hey Mate,

I checked the java.env-file and it contains:




SERVER_JVMFLAGS="-Djava.security.auth.login.config=/opt/zookeeper-cluster/zookeeper/conf/jaas.conf"

which is exactly the place where the pasted jaas.conf is placed.

I also just changed the config to be saslLoginContext and added the
missing semicolon.

But I still get the same exception.

Why would configuring quorum-auth also enable client-server-auth?

Thanks

Sebastian


On 13-Feb-20 5:50 AM, Szalay-Bekő Máté wrote:

Hi Sebastian,

thanks for the more details!

One thing I found in your config is that you should use:
quorum.auth.learner.saslLoginContext=QuorumLearner
quorum.auth.server.saslLoginContext=QuorumServer

so instead of  loginContext, use  saslLoginContext in both lines.  I

found

this in the source code, I think the wiki is wrong (I will fix it

later).

However, actually this didn't really change anything, as the default

values

are anyway
QuorumLearner and  QuorumServer, so you can even skip these lines from

the

config.

I think Rakesh is right, you are seeing exceptions related to not the
QuorumSasl, but the ClientSasl. This is why ZooKeeper tries to find the
'Server' section (what is configuring the server during the

client-server

authentication). The name of this section can be overwritten by the
"zookeeper.sasl.serverconfig" system property.

Based on the exception, ZooKeeper can not find the 'Server' section in
the /opt/zookeeper-cluster/zookeeper/conf/jaas.conf file. Are you sure

this

is the correct jaas.conf? Does the ZooKeeper process have the

permissions

to open this file? You can specify the jaas config file path for

ZooKeeper

by providing custom system property e.g. by exporting
SERVER_JVMFLAGS="-Djava.security.auth.login.config=/path/to/jaas.conf"
before starting zkServer.sh

Also in the jaas.conf you copied here, you are missing a semicolon from

the

end of the last line in the Server block. I am not sure if it is

causing

any parsing error, but I always add the semicolon to the end of the

last

line in the block.

Mate

On Tue, Feb 11, 2020 at 7:53 PM Sebastian Schmitz <
sebastian.schm...@propellerhead.co.nz> wrote:


Hello Rakesh,

as mentioned in the other mail adding the "Server"to jaas.conf didn't

help.

Here are the Configs and Logs (with the Server-part included):

jaas.conf:
QuorumServer {
  org.apache.zookeeper.server.auth.DigestLoginModule required
  user_zookeeper="test";
};

QuorumClient {
  org.apache.zookeeper.server.auth.DigestLoginModule required
  username="zookeeper"
  password="test";
};

Server {
  org.apache.zookeeper.server.auth.DigestLoginModule required
  user_zookeeper="test"
};

Client {
  org.apache.zookeeper.server.auth.DigestLoginModule required
  username="zookeeper"
  password="test";
};

zoo.cfg:
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/mnt/zk_data
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
#


http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance

#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1
dataLogDir=/mnt/zk_data_log
autopurge.snapRetai

Re: Enabling Auth between Zookeeper Servers

2020-02-13 Thread Sebastian Schmitz

Hey Mate,

I checked the java.env-file and it contains:

SERVER_JVMFLAGS="-Djava.security.auth.login.config=/opt/zookeeper-cluster/zookeeper/conf/jaas.conf"

which is exactly the place where the pasted jaas.conf is placed.

I also just changed the config to be saslLoginContext and added the 
missing semicolon.


But I still get the same exception.

Why would configuring quorum-auth also enable client-server-auth?

Thanks

Sebastian


On 13-Feb-20 5:50 AM, Szalay-Bekő Máté wrote:

Hi Sebastian,

thanks for the more details!

One thing I found in your config is that you should use:
quorum.auth.learner.saslLoginContext=QuorumLearner
quorum.auth.server.saslLoginContext=QuorumServer

so instead of  loginContext, use  saslLoginContext in both lines.  I found
this in the source code, I think the wiki is wrong (I will fix it later).
However, actually this didn't really change anything, as the default values
are anyway
QuorumLearner and  QuorumServer, so you can even skip these lines from the
config.

I think Rakesh is right, you are seeing exceptions related to not the
QuorumSasl, but the ClientSasl. This is why ZooKeeper tries to find the
'Server' section (what is configuring the server during the client-server
authentication). The name of this section can be overwritten by the
"zookeeper.sasl.serverconfig" system property.

Based on the exception, ZooKeeper can not find the 'Server' section in
the /opt/zookeeper-cluster/zookeeper/conf/jaas.conf file. Are you sure this
is the correct jaas.conf? Does the ZooKeeper process have the permissions
to open this file? You can specify the jaas config file path for ZooKeeper
by providing custom system property e.g. by exporting
SERVER_JVMFLAGS="-Djava.security.auth.login.config=/path/to/jaas.conf"
before starting zkServer.sh

Also in the jaas.conf you copied here, you are missing a semicolon from the
end of the last line in the Server block. I am not sure if it is causing
any parsing error, but I always add the semicolon to the end of the last
line in the block.

Mate

On Tue, Feb 11, 2020 at 7:53 PM Sebastian Schmitz <
sebastian.schm...@propellerhead.co.nz> wrote:


Hello Rakesh,

as mentioned in the other mail adding the "Server"to jaas.conf didn't help.

Here are the Configs and Logs (with the Server-part included):

jaas.conf:
QuorumServer {
 org.apache.zookeeper.server.auth.DigestLoginModule required
 user_zookeeper="test";
};

QuorumClient {
 org.apache.zookeeper.server.auth.DigestLoginModule required
 username="zookeeper"
 password="test";
};

Server {
 org.apache.zookeeper.server.auth.DigestLoginModule required
 user_zookeeper="test"
};

Client {
 org.apache.zookeeper.server.auth.DigestLoginModule required
 username="zookeeper"
 password="test";
};

zoo.cfg:
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/mnt/zk_data
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
#
http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1
dataLogDir=/mnt/zk_data_log
autopurge.snapRetainCount=3
autopurge.purgeInterval=24
quorum.auth.enableSasl=true
quorum.auth.learnerRequireSasl=false
quorum.auth.serverRequireSasl=false
quorum.auth.learner.loginContext=QuorumLearner
quorum.auth.server.loginContext=QuorumServer
quorum.cnxn.threads.size=20
authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
secureClientPort=2281
server.1=0.0.0.0:2888:3888
server.2=kafkad02.x.azure.com:2888:3888
server.3=kafkad03.x.azure.com:2888:3888

Server-Log:
Using config: /opt/zookeeper-cluster/zookeeper/bin/../conf/zoo.cfg
Feb 11, 2020 18:43:53 + [1 1] com.newrelic INFO: New Relic Agent:
Loading configuration file "/opt/zookeeper-cluster/newrelic/./newrelic.yml"
Feb 11, 2020 18:43:53 + [1 1] com.newrelic INFO: Using default
collector host: collector.newrelic.com
Feb 11, 2020 18:43:53 + [1 1] com.newrelic INFO: New Relic Agent:
Writing to log file:
/opt/zookeeper-cluster/newrelic/logs/newrelic_agent.log
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by
com.newrelic.weave.weavepackage.NewCl

Re: Enabling Auth between Zookeeper Servers

2020-02-11 Thread Sebastian Schmitz
ation section named 'Server' was 
foundin '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf

'.
    at 
org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:211)
    at 
org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:82)
    at 
org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:133)
    at 
org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:114)
    at 
org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:81) 



Best regards

Sebastian


On 12-Feb-20 2:36 AM, Rakesh Radhakrishnan wrote:

>>>>>java.io.IOException: No JAAS configuration section named 'Server'

I could see you have enabled client-server authentication as well. It 
looks to me that the error is coming from that. Please share the 
complete error logs to trace it.
Have you configured "*Server*" section along with the "*QuorumServer*" 
and "*QuorumClient*" sections? If not, please configure "*Server*" 
section along with others and try it out.


Reference: 
https://cwiki.apache.org/confluence/display/ZOOKEEPER/Client-Server+mutual+authentication

image.png

Thanks,
Rakesh

On Tue, Feb 11, 2020 at 7:26 AM Sebastian Schmitz 
<mailto:sebastian.schm...@propellerhead.co.nz>> wrote:


Hello,

I'm currently looking into enabling the Auth between
Zookeeper-Servers
and found this documentation:


https://cwiki.apache.org/confluence/display/ZOOKEEPER/Server-Server+mutual+authentication

However, when I use the config from the document (for Digest-MD5)
I get
this exception in Zookeeper 3.4.14 and also 3.5.6, which I tried
because
I thought using latest version could help:
java.io.IOException: No JAAS configuration section named 'Server' was
found in '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf

And of course that's right, because there's only QuorumServer and
QuorumClient in the jaas.conf:

jaas.conf:
QuorumServer {
    org.apache.zookeeper.server.auth.DigestLoginModule required
    user_zookeeper="test";
};

QuorumClient {
    org.apache.zookeeper.server.auth.DigestLoginModule required
    username="zookeeper"
    password="test";
};

I also tried renaming the QuorumServer to just "Server". No change.

My zoo.cfg:
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/mnt/zk_data
clientPort=2181
dataLogDir=/mnt/zk_data_log
autopurge.snapRetainCount=3
autopurge.purgeInterval=24
quorum.auth.enableSasl=true
quorum.auth.learnerRequireSasl=false
quorum.auth.serverRequireSasl=false
quorum.auth.learner.loginContext=QuorumLearner
quorum.auth.server.loginContext=QuorumServer
quorum.cnxn.threads.size=20
authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
secureClientPort=2281
server.1=0.0.0.0:2888:3888

Any idea what I could try? Or maybe there's some better document
on how
to achieve this?

Thank you

Sebastian


-- 
DISCLAIMER

This email contains information that is confidential and which
may be
legally privileged. If you have received this email in error please

notify the sender immediately and delete the email.
This email is intended
solely for the use of the intended recipient and you may not use or
disclose this email in any way.



--
DISCLAIMER
This email contains information that is confidential and which 
may be 
legally privileged. If you have received this email in error please 


notify the sender immediately and delete the email.
This email is intended 
solely for the use of the intended recipient and you may not use or 
disclose this email in any way. 


Re: Enabling Auth between Zookeeper Servers

2020-02-11 Thread Sebastian Schmitz
59,749 [myid:1] - INFO  [main:QuorumPeerMain@130] - 
Starting quorum peer
2020-02-11 18:43:59,788 [myid:1] - INFO [main:ServerCnxnFactory@117] - 
Using org.apache.zookeeper.server.NIOServerCnxnFactory as server 
connection factory
2020-02-11 18:43:59,804 [myid:1] - INFO 
[PurgeTask:DatadirCleanupManager$PurgeTask@144] - Purge task completed.
'.20-02-11 18:43:59,826 [myid:1] - ERROR [main:ServerCnxnFactory@210] - 
No JAAS configuration section named 'Server' was foundin 
'/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
2020-02-11 18:43:59,827 [myid:1] - ERROR [main:QuorumPeerMain@92] - 
Unexpected exception, exiting abnormally
java.io.IOException: No JAAS configuration section named 'Server' was 
foundin '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf

'.
    at 
org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:211)
    at 
org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:82)
    at 
org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:133)
    at 
org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:114)
    at 
org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:81)


Best regards

Sebastian

On 11-Feb-20 9:56 PM, Szalay-Bekő Máté wrote:

Hello Sebastian,

In general I think you did configure ZooKeeper just fine. A few remarks:
- I am not sure how ZooKeeper server-server authentication is expected to
work when you only use a single server. Would you mind trying to start a
Quorum with e.g. 3 servers?
- also, I think it is a good idea to avoid using 0.0.0.0 as hostname,
especially if you would run the cluster on multiple different servers /
docker containers. Try using the fully qualified domain name for a
multi-server setup, or if you just test multiple ZooKeeper servers on the
same machine, then just use 127.0.0.1. (maybe it has no effect in the
current case, but for SSL or for dynamic reconfig it might be bad to use
0.0.0.0. Also I remember problems with rolling restarts when using 0.0.0.0
in the config)
- is there a reason why you set 'quorum.auth.learnerRequireSasl' and
'quorum.auth.serverRequireSasl' to false? Using false is usually good idea
during rolling upgrade, but if you start a new cluster and want to use
server-server authentication, then you can just set them to true.

I don't understand why you got the exception " No JAAS configuration
section named 'Server' was found" ... Setting the loginContext should have
fixed that. If you still see the same issue with the 3 server setup, then
can you please share the config files, the command how you start ZooKeeper
and also the log files with us, so that we can look deeper?

Kind regards,
Mate

On Tue, Feb 11, 2020 at 2:56 AM Sebastian Schmitz <
sebastian.schm...@propellerhead.co.nz> wrote:


Hello,

I'm currently looking into enabling the Auth between Zookeeper-Servers
and found this documentation:


https://cwiki.apache.org/confluence/display/ZOOKEEPER/Server-Server+mutual+authentication

However, when I use the config from the document (for Digest-MD5) I get
this exception in Zookeeper 3.4.14 and also 3.5.6, which I tried because
I thought using latest version could help:
java.io.IOException: No JAAS configuration section named 'Server' was
found in '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf

And of course that's right, because there's only QuorumServer and
QuorumClient in the jaas.conf:

jaas.conf:
QuorumServer {
 org.apache.zookeeper.server.auth.DigestLoginModule required
 user_zookeeper="test";
};

QuorumClient {
 org.apache.zookeeper.server.auth.DigestLoginModule required
 username="zookeeper"
 password="test";
};

I also tried renaming the QuorumServer to just "Server". No change.

My zoo.cfg:
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/mnt/zk_data
clientPort=2181
dataLogDir=/mnt/zk_data_log
autopurge.snapRetainCount=3
autopurge.purgeInterval=24
quorum.auth.enableSasl=true
quorum.auth.learnerRequireSasl=false
quorum.auth.serverRequireSasl=false
quorum.auth.learner.loginContext=QuorumLearner
quorum.auth.server.loginContext=QuorumServer
quorum.cnxn.threads.size=20
authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
secureClientPort=2281
server.1=0.0.0.0:2888:3888

Any idea what I could try? Or maybe there's some better document on how
to achieve this?

Thank you

Sebastian


--
DISCLAIMER
This email contains information that is confidential and which
may be
legally privileged. If you have received this email in error please

notify the sender immediately and delete the email.
This email is intended
solely for the use of the intended recipient and you may not use or
disclose this email in any way.



--
DISCLAIMER
This email contains information that is confidential and which 
may be 
legally privileged. If you have received this email in er

Enabling Auth between Zookeeper Servers

2020-02-10 Thread Sebastian Schmitz

Hello,

I'm currently looking into enabling the Auth between Zookeeper-Servers 
and found this documentation:


https://cwiki.apache.org/confluence/display/ZOOKEEPER/Server-Server+mutual+authentication

However, when I use the config from the document (for Digest-MD5) I get 
this exception in Zookeeper 3.4.14 and also 3.5.6, which I tried because 
I thought using latest version could help:
java.io.IOException: No JAAS configuration section named 'Server' was 
found in '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf


And of course that's right, because there's only QuorumServer and 
QuorumClient in the jaas.conf:


jaas.conf:
QuorumServer {
   org.apache.zookeeper.server.auth.DigestLoginModule required
   user_zookeeper="test";
};

QuorumClient {
   org.apache.zookeeper.server.auth.DigestLoginModule required
   username="zookeeper"
   password="test";
};

I also tried renaming the QuorumServer to just "Server". No change.

My zoo.cfg:
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/mnt/zk_data
clientPort=2181
dataLogDir=/mnt/zk_data_log
autopurge.snapRetainCount=3
autopurge.purgeInterval=24
quorum.auth.enableSasl=true
quorum.auth.learnerRequireSasl=false
quorum.auth.serverRequireSasl=false
quorum.auth.learner.loginContext=QuorumLearner
quorum.auth.server.loginContext=QuorumServer
quorum.cnxn.threads.size=20
authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
secureClientPort=2281
server.1=0.0.0.0:2888:3888

Any idea what I could try? Or maybe there's some better document on how 
to achieve this?


Thank you

Sebastian


--
DISCLAIMER
This email contains information that is confidential and which 
may be 
legally privileged. If you have received this email in error please 


notify the sender immediately and delete the email.
This email is intended 
solely for the use of the intended recipient and you may not use or 
disclose this email in any way. 


Re: Zookeeper 3.5.6 - Exception causing close of session 0x0: null

2019-10-29 Thread Sebastian Schmitz

Hello Enrico,

yes, a custom made client, that was updated and now is stopped again...

The difference is that in new version only this is the output:

Oct 25 10:56:23   zookeeper_node_1: 2019-10-24 21:56:23,119 [myid:1] - 
WARN  [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of 
session 0x0: null


In the 3.4.14 it was also stating where the connection came from before 
giving the warning:


Oct 21 16:02:16 zookeeper_node_2: 2019-10-2103:02:16,679 [myid:2] - INFO 
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@222] - 
Accepted socket connection from /10.24.8.13:60517


Best regards

Sebastian


On 29-Oct-19 7:06 PM, Enrico Olivelli wrote:

Il mar 29 ott 2019, 00:52 Sebastian Schmitz <
sebastian.schm...@propellerhead.co.nz> ha scritto:


Hello again,

just as a final update on this. We had a weird consumer which
someone updated without informing me...


Can you please define 'weird'? A custom made client?



Turning it off also turned off

the errors from spamming the logs.

However it seems like the 3.5.6 is less informative than the 3.4.14 we
had before as 3.4.14 also had the IP-Address when the connection was
opened in the messages and not just the warnings...


If you can point to the differences we can bring back precious logs

Great to see you found the problem

Enrico


Best regards

Sebastian


On 25-Oct-19 1:26 PM, Sebastian Schmitz wrote:

Hello Enrico,

thanks for the reply. I'll try to find the exception, there was none
around 14:50 when it first appeared. I also try to find the client
that's connecting just before that as it's not a Kafka-node:

Oct 21 14:50:31 zookeeper_node_3: 2019-10-21 01:50:31,532 [myid:3] -
INFO
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@222] -
Accepted socket connection from /10.24.8.13:64261
Oct 21 14:50:31 zookeeper_node_3: 2019-10-21 01:50:31,544 [myid:3] -
WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@383] -
Exception causing close of session 0x0: null
Oct 21 14:50:31 zookeeper_node_3: 2019-10-21 01:50:31,545 [myid:3] -
INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1056] -
Closed socket connection for client /10.24.8.13:64261 (no session
established for client)

The same IP is able to connect to Zookeeper successfully some time
before:

Oct 21 13:18:44 zookeeper_node_3: 2019-10-21 00:18:44,823 [myid:3] -
INFO
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@222] -
Accepted socket connection from /10.24.8.13:50212
Oct 21 13:18:44 zookeeper_node_3: 2019-10-21 00:18:44,825 [myid:3] -
INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@949] -
Client attempting to establish new session at /10.24.8.13:50212
Oct 21 13:18:44 zookeeper_node_3: 2019-10-21 00:18:44,833 [myid:3] -
INFO  [CommitProcessor:3:ZooKeeperServer@694] - Established session
0x30b2dfdd8ac with negotiated timeout 1 for client
/10.24.8.13:50212
Oct 21 13:18:53 zookeeper_node_3: 2019-10-21 00:18:53,374 [myid:3] -
INFO
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@222] -
Accepted socket connection from /10.24.8.13:50314
Oct 21 13:18:53 zookeeper_node_3: 2019-10-21 00:18:53,374 [myid:3] -
INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@949] -
Client attempting to establish new session at /10.24.8.13:50314
Oct 21 13:18:53 zookeeper_node_3: 2019-10-21 00:18:53,385 [myid:3] -
INFO  [CommitProcessor:3:ZooKeeperServer@694] - Established session
0x30b2dfdd8ac0001 with negotiated timeout 1 for client
/10.24.8.13:50314
Oct 21 13:18:53 zookeeper_node_3: 2019-10-21 00:18:53,588 [myid:3] -
INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1056] -
Closed socket connection for client /10.24.8.13:50314 which had
sessionid 0x30b2dfdd8ac0001

I also checked the Logs from the update I did on Wednesday night to
update to 3.5.6. This is the Log of the start of zookeeper_node_1
until the errors show up... Can't see any exceptions :(

Oct 23 02:07:03 zookeeper_node_1: setting ZOO_LOG4J_PROP=INFO,CONSOLE
Oct 23 02:07:03 zookeeper_node_1: ZooKeeper JMX enabled by default
Oct 23 02:07:03 zookeeper_node_1: Using config:
/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../conf/zoo.cfg
Oct 23 02:07:05 zookeeper_node_1: Oct 22, 2019 13:07:05 + [1 1]
com.newrelic INFO: New Relic Agent: Loading configuration file
"/opt/zookeeper-cluster/newrelic/./newrelic.yml"
Oct 23 02:07:05 zookeeper_node_1: Oct 22, 2019 13:07:05 + [1 1]
com.newrelic INFO: Using default collector host: collector.newrelic.com
Oct 23 02:07:05 zookeeper_node_1: Oct 22, 2019 13:07:05 + [1 1]
com.newrelic INFO: New Relic Agent: Writing to log file:
/opt/zookeeper-cluster/newrelic/logs/newrelic_agent.log
Oct 23 02:07:28 zookeeper_node_1: 2019-10-22 13:07:28,743 [myid:] -
INFO  [main:QuorumPeerConfig@133] - Reading configuration from:
/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../conf/zoo.cfg
Oct 23 02:07:28 zookeeper_node_1: 2019-10-22 13:07:28,76

Re: Zookeeper 3.5.6 - Exception causing close of session 0x0: null

2019-10-28 Thread Sebastian Schmitz

Hello again,

just as a final update on this. We had a weird consumer which 
someone updated without informing me... Turning it off also turned off 
the errors from spamming the logs.


However it seems like the 3.5.6 is less informative than the 3.4.14 we 
had before as 3.4.14 also had the IP-Address when the connection was 
opened in the messages and not just the warnings...


Best regards

Sebastian


On 25-Oct-19 1:26 PM, Sebastian Schmitz wrote:

Hello Enrico,

thanks for the reply. I'll try to find the exception, there was none 
around 14:50 when it first appeared. I also try to find the client 
that's connecting just before that as it's not a Kafka-node:


Oct 21 14:50:31 zookeeper_node_3: 2019-10-21 01:50:31,532 [myid:3] - 
INFO 
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@222] - 
Accepted socket connection from /10.24.8.13:64261
Oct 21 14:50:31 zookeeper_node_3: 2019-10-21 01:50:31,544 [myid:3] - 
WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@383] - 
Exception causing close of session 0x0: null
Oct 21 14:50:31 zookeeper_node_3: 2019-10-21 01:50:31,545 [myid:3] - 
INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1056] - 
Closed socket connection for client /10.24.8.13:64261 (no session 
established for client)


The same IP is able to connect to Zookeeper successfully some time 
before:


Oct 21 13:18:44 zookeeper_node_3: 2019-10-21 00:18:44,823 [myid:3] - 
INFO 
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@222] - 
Accepted socket connection from /10.24.8.13:50212
Oct 21 13:18:44 zookeeper_node_3: 2019-10-21 00:18:44,825 [myid:3] - 
INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@949] - 
Client attempting to establish new session at /10.24.8.13:50212
Oct 21 13:18:44 zookeeper_node_3: 2019-10-21 00:18:44,833 [myid:3] - 
INFO  [CommitProcessor:3:ZooKeeperServer@694] - Established session 
0x30b2dfdd8ac with negotiated timeout 1 for client 
/10.24.8.13:50212
Oct 21 13:18:53 zookeeper_node_3: 2019-10-21 00:18:53,374 [myid:3] - 
INFO 
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@222] - 
Accepted socket connection from /10.24.8.13:50314
Oct 21 13:18:53 zookeeper_node_3: 2019-10-21 00:18:53,374 [myid:3] - 
INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@949] - 
Client attempting to establish new session at /10.24.8.13:50314
Oct 21 13:18:53 zookeeper_node_3: 2019-10-21 00:18:53,385 [myid:3] - 
INFO  [CommitProcessor:3:ZooKeeperServer@694] - Established session 
0x30b2dfdd8ac0001 with negotiated timeout 1 for client 
/10.24.8.13:50314
Oct 21 13:18:53 zookeeper_node_3: 2019-10-21 00:18:53,588 [myid:3] - 
INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1056] - 
Closed socket connection for client /10.24.8.13:50314 which had 
sessionid 0x30b2dfdd8ac0001


I also checked the Logs from the update I did on Wednesday night to 
update to 3.5.6. This is the Log of the start of zookeeper_node_1 
until the errors show up... Can't see any exceptions :(


Oct 23 02:07:03 zookeeper_node_1: setting ZOO_LOG4J_PROP=INFO,CONSOLE
Oct 23 02:07:03 zookeeper_node_1: ZooKeeper JMX enabled by default
Oct 23 02:07:03 zookeeper_node_1: Using config: 
/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../conf/zoo.cfg
Oct 23 02:07:05 zookeeper_node_1: Oct 22, 2019 13:07:05 + [1 1] 
com.newrelic INFO: New Relic Agent: Loading configuration file 
"/opt/zookeeper-cluster/newrelic/./newrelic.yml"
Oct 23 02:07:05 zookeeper_node_1: Oct 22, 2019 13:07:05 + [1 1] 
com.newrelic INFO: Using default collector host: collector.newrelic.com
Oct 23 02:07:05 zookeeper_node_1: Oct 22, 2019 13:07:05 + [1 1] 
com.newrelic INFO: New Relic Agent: Writing to log file: 
/opt/zookeeper-cluster/newrelic/logs/newrelic_agent.log
Oct 23 02:07:28 zookeeper_node_1: 2019-10-22 13:07:28,743 [myid:] - 
INFO  [main:QuorumPeerConfig@133] - Reading configuration from: 
/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../conf/zoo.cfg
Oct 23 02:07:28 zookeeper_node_1: 2019-10-22 13:07:28,764 [myid:] - 
INFO  [main:QuorumPeerConfig@385] - clientPortAddress is 
0.0.0.0/0.0.0.0:2181
Oct 23 02:07:28 zookeeper_node_1: 2019-10-22 13:07:28,764 [myid:] - 
INFO  [main:QuorumPeerConfig@389] - secureClientPort is not set
Oct 23 02:07:28 zookeeper_node_1: 2019-10-22 13:07:28,877 [myid:1] - 
INFO  [main:DatadirCleanupManager@78] - autopurge.snapRetainCount set 
to 3
Oct 23 02:07:28 zookeeper_node_1: 2019-10-22 13:07:28,878 [myid:1] - 
INFO  [main:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0
Oct 23 02:07:28 zookeeper_node_1: 2019-10-22 13:07:28,878 [myid:1] - 
INFO  [main:DatadirCleanupManager@101] - Purge task is not scheduled.
Oct 23 02:07:28 zookeeper_node_1: 2019-10-22 13:07:28,889 [myid:1] - 
INFO  [main:ManagedUtil@46] - Log4j found with jmx enabled.
Oct 23 02:07:29 zookeeper_node_1: 2019-10-22 13:07:29,046 [myid:1] - 
INFO  [main:QuorumPeerMain@141] - Starting quorum peer
Oct 2

Re: Zookeeper 3.5.6 - Exception causing close of session 0x0: null

2019-10-24 Thread Sebastian Schmitz
f 
session 0x0: null
Oct 23 02:07:46 zookeeper_node_1: 2019-10-22 13:07:46,474 [myid:1] - 
WARN  [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of 
session 0x0: null
Oct 23 02:07:47 zookeeper_node_1: 2019-10-22 13:07:47,111 [myid:1] - 
WARN  [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close of 
session 0x0: null
Oct 23 02:07:47 zookeeper_node_1: 2019-10-22 13:07:47,292 [myid:1] - 
WARN  [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of 
session 0x0: null


Thanks

Sebastian

On 25-Oct-19 11:17 AM, Enrico Olivelli wrote:

Sebastian
This 'null' looks like a NullPointerException (this kind f exceptions
tracks only the full stacktrace at the first occurrence usually)
Can you find some full stacktrace of the error?
Are you able to enable more detailed logging?

Do you see errors on the client (kafka broker?) ? Maybe on
oeg.apache.zookeeper.xxx logs

Enrico

Il ven 25 ott 2019, 00:05 Sebastian Schmitz <
sebastian.schm...@propellerhead.co.nz> ha scritto:


Hello,

I did a rolling upgrade of the Zookeeper from 3.4.14 to 3.5.6 on 23rd in
my Test-Environment. It's zookeeping a three node Kafka-Cluster running
on 2.3.0.

After the update I get spammed with these warnings:

Oct 25 10:56:23   zookeeper_node_1: 2019-10-24 21:56:23,119 [myid:1] -
WARN  [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of
session 0x0: null
Oct 25 10:56:23   zookeeper_node_2: 2019-10-24 21:56:23,478 [myid:2] -
WARN  [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of
session 0x0: null
Oct 25 10:56:23   zookeeper_node_3: 2019-10-24 21:56:23,531 [myid:3] -
WARN  [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of
session 0x0: null
Oct 25 10:56:23   zookeeper_node_1: 2019-10-24 21:56:23,776 [myid:1] -
WARN  [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close of
session 0x0: null
Oct 25 10:56:23   zookeeper_node_2: 2019-10-24 21:56:23,879 [myid:2] -
WARN  [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of
session 0x0: null
Oct 25 10:56:24   zookeeper_node_3: 2019-10-24 21:56:24,086 [myid:3] -
WARN  [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of
session 0x0: null
Oct 25 10:56:24   zookeeper_node_1: 2019-10-24 21:56:24,090 [myid:1] -
WARN  [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of
session 0x0: null
Oct 25 10:56:24   zookeeper_node_2: 2019-10-24 21:56:24,497 [myid:2] -
WARN  [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of
session 0x0: null
Oct 25 10:56:24   zookeeper_node_3: 2019-10-24 21:56:24,601 [myid:3] -
WARN  [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of
session 0x0: null
Oct 25 10:56:24   zookeeper_node_1: 2019-10-24 21:56:24,798 [myid:1] -
WARN  [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of
session 0x0: null
Oct 25 10:56:24   zookeeper_node_3: 2019-10-24 21:56:24,904 [myid:3] -
WARN  [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of
session 0x0: null
Oct 25 10:56:25   zookeeper_node_2: 2019-10-24 21:56:25,007 [myid:2] -
WARN  [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of
session 0x0: null
Oct 25 10:56:25   zookeeper_node_1: 2019-10-24 21:56:25,260 [myid:1] -
WARN  [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of
session 0x0: null
Oct 25 10:56:25   zookeeper_node_3: 2019-10-24 21:56:25,467 [myid:3] -
WARN  [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close of
session 0x0: null
Oct 25 10:56:25   zookeeper_node_2: 2019-10-24 21:56:25,619 [myid:2] -
WARN  [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close of
session 0x0: null
Oct 25 10:56:25   zookeeper_node_1: 2019-10-24 21:56:25,666 [myid:1] -
WARN  [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of
session 0x0: null
Oct 25 10:56:26   zookeeper_node_3: 2019-10-24 21:56:25,923 [myid:3] -
WARN  [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of
session 0x0: null
Oct 25 10:56:26   zookeeper_node_2: 2019-10-24 21:56:26,178 [myid:2] -
WARN  [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of
session 0x0: null
Oct 25 10:56:26   zookeeper_node_1: 2019-10-24 21:56:26,329 [myid:1] -
WARN  [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of
session 0x0: null
Oct 25 10:56:26   zookeeper_node_3: 2019-10-24 21:56:26,484 [myid:3] -
WARN  [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of
session 0x0: null
Oct 25 10:56:26   zookeeper_node_2: 2019-10-24 21:56:26,792 [myid:2] -
WARN  [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of
session 0x0: null
Oct 25 10:56:26   zookeeper_node_3: 2019-10-24 21:56:26,890 [myid:3] -
WARN  [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of
session 0x0: null
Oct 25 10:56:26   zookeeper_node_1: 2019-10-24 21:56:26,888 [myid:1] -
WARN  [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of
session 0x0: null
Oct 25 10:56:27   zookeeper_node_2: 

Re: Zookeeper 3.5.6 - Exception causing close of session 0x0: null

2019-10-24 Thread Sebastian Schmitz

Hello Again,

I have a small correction to make... It didn't start after the upgrade 
to 3.5.6, but just in the middle of a day at 14:50 on 21st And the 
upgrade to 3.5.6 didn't automagically solve it


So any help to find the cause of this would be great.

Thanks

Sebastian


On 25-Oct-19 11:05 AM, Sebastian Schmitz wrote:

Hello,

I did a rolling upgrade of the Zookeeper from 3.4.14 to 3.5.6 on 23rd 
in my Test-Environment. It's zookeeping a three node Kafka-Cluster 
running on 2.3.0.


After the update I get spammed with these warnings:

Oct 25 10:56:23   zookeeper_node_1: 2019-10-24 21:56:23,119 [myid:1] - 
WARN  [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close 
of session 0x0: null
Oct 25 10:56:23   zookeeper_node_2: 2019-10-24 21:56:23,478 [myid:2] - 
WARN  [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close 
of session 0x0: null
Oct 25 10:56:23   zookeeper_node_3: 2019-10-24 21:56:23,531 [myid:3] - 
WARN  [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close 
of session 0x0: null
Oct 25 10:56:23   zookeeper_node_1: 2019-10-24 21:56:23,776 [myid:1] - 
WARN  [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close 
of session 0x0: null
Oct 25 10:56:23   zookeeper_node_2: 2019-10-24 21:56:23,879 [myid:2] - 
WARN  [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close 
of session 0x0: null
Oct 25 10:56:24   zookeeper_node_3: 2019-10-24 21:56:24,086 [myid:3] - 
WARN  [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close 
of session 0x0: null
Oct 25 10:56:24   zookeeper_node_1: 2019-10-24 21:56:24,090 [myid:1] - 
WARN  [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close 
of session 0x0: null
Oct 25 10:56:24   zookeeper_node_2: 2019-10-24 21:56:24,497 [myid:2] - 
WARN  [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close 
of session 0x0: null
Oct 25 10:56:24   zookeeper_node_3: 2019-10-24 21:56:24,601 [myid:3] - 
WARN  [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close 
of session 0x0: null
Oct 25 10:56:24   zookeeper_node_1: 2019-10-24 21:56:24,798 [myid:1] - 
WARN  [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close 
of session 0x0: null
Oct 25 10:56:24   zookeeper_node_3: 2019-10-24 21:56:24,904 [myid:3] - 
WARN  [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close 
of session 0x0: null
Oct 25 10:56:25   zookeeper_node_2: 2019-10-24 21:56:25,007 [myid:2] - 
WARN  [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close 
of session 0x0: null
Oct 25 10:56:25   zookeeper_node_1: 2019-10-24 21:56:25,260 [myid:1] - 
WARN  [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close 
of session 0x0: null
Oct 25 10:56:25   zookeeper_node_3: 2019-10-24 21:56:25,467 [myid:3] - 
WARN  [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close 
of session 0x0: null
Oct 25 10:56:25   zookeeper_node_2: 2019-10-24 21:56:25,619 [myid:2] - 
WARN  [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close 
of session 0x0: null
Oct 25 10:56:25   zookeeper_node_1: 2019-10-24 21:56:25,666 [myid:1] - 
WARN  [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close 
of session 0x0: null
Oct 25 10:56:26   zookeeper_node_3: 2019-10-24 21:56:25,923 [myid:3] - 
WARN  [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close 
of session 0x0: null
Oct 25 10:56:26   zookeeper_node_2: 2019-10-24 21:56:26,178 [myid:2] - 
WARN  [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close 
of session 0x0: null
Oct 25 10:56:26   zookeeper_node_1: 2019-10-24 21:56:26,329 [myid:1] - 
WARN  [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close 
of session 0x0: null
Oct 25 10:56:26   zookeeper_node_3: 2019-10-24 21:56:26,484 [myid:3] - 
WARN  [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close 
of session 0x0: null
Oct 25 10:56:26   zookeeper_node_2: 2019-10-24 21:56:26,792 [myid:2] - 
WARN  [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close 
of session 0x0: null
Oct 25 10:56:26   zookeeper_node_3: 2019-10-24 21:56:26,890 [myid:3] - 
WARN  [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close 
of session 0x0: null
Oct 25 10:56:26   zookeeper_node_1: 2019-10-24 21:56:26,888 [myid:1] - 
WARN  [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close 
of session 0x0: null
Oct 25 10:56:27   zookeeper_node_2: 2019-10-24 21:56:27,347 [myid:2] - 
WARN  [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close 
of session 0x0: null
Oct 25 10:56:27   zookeeper_node_1: 2019-10-24 21:56:27,354 [myid:1] - 
WARN  [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close 
of session 0x0: null
Oct 25 10:56:27   zookeeper_node_3: 2019-10-24 21:56:27,662 [myid:3] - 
WARN  [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close 
of session 0x0: null
Oct 25 10:56:27   zookeeper_node_3: 2019-10-24 21:56:27,810 [myid:3] - 
WARN  [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close 
of session 0x0: null
Oct 25 10:56

Zookeeper 3.5.6 - Exception causing close of session 0x0: null

2019-10-24 Thread Sebastian Schmitz

Hello,

I did a rolling upgrade of the Zookeeper from 3.4.14 to 3.5.6 on 23rd in 
my Test-Environment. It's zookeeping a three node Kafka-Cluster running 
on 2.3.0.


After the update I get spammed with these warnings:

Oct 25 10:56:23   zookeeper_node_1: 2019-10-24 21:56:23,119 [myid:1] - 
WARN  [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of 
session 0x0: null
Oct 25 10:56:23   zookeeper_node_2: 2019-10-24 21:56:23,478 [myid:2] - 
WARN  [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of 
session 0x0: null
Oct 25 10:56:23   zookeeper_node_3: 2019-10-24 21:56:23,531 [myid:3] - 
WARN  [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of 
session 0x0: null
Oct 25 10:56:23   zookeeper_node_1: 2019-10-24 21:56:23,776 [myid:1] - 
WARN  [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close of 
session 0x0: null
Oct 25 10:56:23   zookeeper_node_2: 2019-10-24 21:56:23,879 [myid:2] - 
WARN  [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of 
session 0x0: null
Oct 25 10:56:24   zookeeper_node_3: 2019-10-24 21:56:24,086 [myid:3] - 
WARN  [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of 
session 0x0: null
Oct 25 10:56:24   zookeeper_node_1: 2019-10-24 21:56:24,090 [myid:1] - 
WARN  [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of 
session 0x0: null
Oct 25 10:56:24   zookeeper_node_2: 2019-10-24 21:56:24,497 [myid:2] - 
WARN  [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of 
session 0x0: null
Oct 25 10:56:24   zookeeper_node_3: 2019-10-24 21:56:24,601 [myid:3] - 
WARN  [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of 
session 0x0: null
Oct 25 10:56:24   zookeeper_node_1: 2019-10-24 21:56:24,798 [myid:1] - 
WARN  [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of 
session 0x0: null
Oct 25 10:56:24   zookeeper_node_3: 2019-10-24 21:56:24,904 [myid:3] - 
WARN  [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of 
session 0x0: null
Oct 25 10:56:25   zookeeper_node_2: 2019-10-24 21:56:25,007 [myid:2] - 
WARN  [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of 
session 0x0: null
Oct 25 10:56:25   zookeeper_node_1: 2019-10-24 21:56:25,260 [myid:1] - 
WARN  [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of 
session 0x0: null
Oct 25 10:56:25   zookeeper_node_3: 2019-10-24 21:56:25,467 [myid:3] - 
WARN  [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close of 
session 0x0: null
Oct 25 10:56:25   zookeeper_node_2: 2019-10-24 21:56:25,619 [myid:2] - 
WARN  [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close of 
session 0x0: null
Oct 25 10:56:25   zookeeper_node_1: 2019-10-24 21:56:25,666 [myid:1] - 
WARN  [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of 
session 0x0: null
Oct 25 10:56:26   zookeeper_node_3: 2019-10-24 21:56:25,923 [myid:3] - 
WARN  [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of 
session 0x0: null
Oct 25 10:56:26   zookeeper_node_2: 2019-10-24 21:56:26,178 [myid:2] - 
WARN  [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of 
session 0x0: null
Oct 25 10:56:26   zookeeper_node_1: 2019-10-24 21:56:26,329 [myid:1] - 
WARN  [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of 
session 0x0: null
Oct 25 10:56:26   zookeeper_node_3: 2019-10-24 21:56:26,484 [myid:3] - 
WARN  [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of 
session 0x0: null
Oct 25 10:56:26   zookeeper_node_2: 2019-10-24 21:56:26,792 [myid:2] - 
WARN  [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of 
session 0x0: null
Oct 25 10:56:26   zookeeper_node_3: 2019-10-24 21:56:26,890 [myid:3] - 
WARN  [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of 
session 0x0: null
Oct 25 10:56:26   zookeeper_node_1: 2019-10-24 21:56:26,888 [myid:1] - 
WARN  [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of 
session 0x0: null
Oct 25 10:56:27   zookeeper_node_2: 2019-10-24 21:56:27,347 [myid:2] - 
WARN  [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of 
session 0x0: null
Oct 25 10:56:27   zookeeper_node_1: 2019-10-24 21:56:27,354 [myid:1] - 
WARN  [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of 
session 0x0: null
Oct 25 10:56:27   zookeeper_node_3: 2019-10-24 21:56:27,662 [myid:3] - 
WARN  [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of 
session 0x0: null
Oct 25 10:56:27   zookeeper_node_3: 2019-10-24 21:56:27,810 [myid:3] - 
WARN  [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of 
session 0x0: null
Oct 25 10:56:27   zookeeper_node_1: 2019-10-24 21:56:27,811 [myid:1] - 
WARN  [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of 
session 0x0: null
Oct 25 10:56:27   zookeeper_node_2: 2019-10-24 21:56:27,816 [myid:2] - 
WARN  [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of 
session 0x0: null
Oct 25 10:56:28