Re: Enabling Auth between Zookeeper Servers
Hi Sebastian! I was able to setup digest authentication, uploaded my results here: https://github.com/symat/zookeeper-docker-test You can see my docker compose file: https://github.com/symat/zookeeper-docker-test/blob/master/3_nodes_digest_quorum_auth.yml also the zoo.cfg template: https://github.com/symat/zookeeper-docker-test/blob/master/conf/digest_zoo.cfg and the jaas.cfg file: https://github.com/symat/zookeeper-docker-test/blob/master/conf/digest_jaas.conf It works for me, using ZooKeeper 3.5.6. Although I haven't follow your config everywhere. Still, I wasn't able to reproduce your exception, only when I actually deleted the jaas config file. Are you sure that the ZooKeeper process in docker can see / open that file? I created a patched ZooKeeper 3.5.6 for you (you can download from here: https://drive.google.com/open?id=1KEPjNkiKf937jMJHAicwW9WATEuyRZIo), where more details are printed in case of errors. E.g. in my case when I deleted the jaas config file, I get: zoo1_1 | 2020-02-14 07:04:33,288 [myid:1] - ERROR [main:ServerCnxnFactory@246] - No JAAS configuration section named 'Server' was found in '/scripts/conf/digest_jaas.conf'. zoo1_1 | java.lang.SecurityException: java.io.IOException: /scripts/conf/digest_jaas.conf (No such file or directory) zoo1_1 | at sun.security.provider.ConfigFile$Spi.(ConfigFile.java:137) zoo1_1 | at sun.security.provider.ConfigFile.(ConfigFile.java:102) zoo1_1 | at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) zoo1_1 | at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) zoo1_1 | at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) zoo1_1 | at java.lang.reflect.Constructor.newInstance(Constructor.java:423) zoo1_1 | at java.lang.Class.newInstance(Class.java:442) zoo1_1 | at javax.security.auth.login.Configuration$2.run(Configuration.java:255) zoo1_1 | at javax.security.auth.login.Configuration$2.run(Configuration.java:247) zoo1_1 | at java.security.AccessController.doPrivileged(Native Method) zoo1_1 | at javax.security.auth.login.Configuration.getConfiguration(Configuration.java:246) zoo1_1 | at org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:210) zoo1_1 | at org.apache.zookeeper.server.NettyServerCnxnFactory.configure(NettyServerCnxnFactory.java:383) zoo1_1 | at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:148) zoo1_1 | at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:123) zoo1_1 | at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82) zoo1_1 | Caused by: java.io.IOException: /scripts/conf/digest_jaas.conf (No such file or directory) zoo1_1 | at sun.security.provider.ConfigFile$Spi.ioException(ConfigFile.java:666) zoo1_1 | at sun.security.provider.ConfigFile$Spi.init(ConfigFile.java:262) zoo1_1 | at sun.security.provider.ConfigFile$Spi.(ConfigFile.java:135) zoo1_1 | ... 15 more z Kind regards, Mate On Fri, Feb 14, 2020 at 7:12 AM sagar shukla wrote: > O > Sent from Yahoo Mail on Android > > On Fri, Feb 14, 2020 at 11:02 AM, Szalay-Bekő Máté< > szalay.beko.m...@gmail.com> wrote: Hi Sebastian, > > > But I still get the same exception. > at this point I don't know why this happen... Adding the Server section to > the jaas config should have helped. Unfortunately the exact exception is > not printed out into the logs, just the error message, so it is hard to > find out more details. > > I will try to reproduce your case with 3.5.6 locally and see if it works. I > never actually used digest authentication before... we always use kerberos > in production. If it works, I will share my configs / dockerfiles and send > you a patched version with more debug info printed out. > > > Why would configuring quorum-auth also enable client-server-auth? > it is not very logical indeed... if I see it right, based on the code once > you set the java.security.auth.login.config property, then ZooKeeper > assumes you want to use server-client sasl authentication. I guess the > quorum-auth feature was added later and they introduced 'enable' config > property for this, but forget to introduce the same config for the client > authentication. I also guess most of the people are interested in the > client authentication and it is rare that someone does't need that but > needs quorum auth. Still, the current behaviour is not good I think. I will > submit a jira ticket requesting an improvement here when I will have time, > but feel free to submit it yourself if you wish. > > Kind regards, > Mate > > On Thu, Feb 13, 2020 at 7:41 PM Sebastian Schmitz < > sebastian.schm...@propellerhead.co.nz> wrote: > > > Hey Mate, > > > > I checked the java.env-file and it contains: > > > > > > >
Re: Enabling Auth between Zookeeper Servers
O
Sent from Yahoo Mail on Android
On Fri, Feb 14, 2020 at 11:02 AM, Szalay-Bekő
Máté wrote: Hi Sebastian,
> But I still get the same exception.
at this point I don't know why this happen... Adding the Server section to
the jaas config should have helped. Unfortunately the exact exception is
not printed out into the logs, just the error message, so it is hard to
find out more details.
I will try to reproduce your case with 3.5.6 locally and see if it works. I
never actually used digest authentication before... we always use kerberos
in production. If it works, I will share my configs / dockerfiles and send
you a patched version with more debug info printed out.
> Why would configuring quorum-auth also enable client-server-auth?
it is not very logical indeed... if I see it right, based on the code once
you set the java.security.auth.login.config property, then ZooKeeper
assumes you want to use server-client sasl authentication. I guess the
quorum-auth feature was added later and they introduced 'enable' config
property for this, but forget to introduce the same config for the client
authentication. I also guess most of the people are interested in the
client authentication and it is rare that someone does't need that but
needs quorum auth. Still, the current behaviour is not good I think. I will
submit a jira ticket requesting an improvement here when I will have time,
but feel free to submit it yourself if you wish.
Kind regards,
Mate
On Thu, Feb 13, 2020 at 7:41 PM Sebastian Schmitz <
sebastian.schm...@propellerhead.co.nz> wrote:
> Hey Mate,
>
> I checked the java.env-file and it contains:
>
>
> SERVER_JVMFLAGS="-Djava.security.auth.login.config=/opt/zookeeper-cluster/zookeeper/conf/jaas.conf"
>
> which is exactly the place where the pasted jaas.conf is placed.
>
> I also just changed the config to be saslLoginContext and added the
> missing semicolon.
>
> But I still get the same exception.
>
> Why would configuring quorum-auth also enable client-server-auth?
>
> Thanks
>
> Sebastian
>
>
> On 13-Feb-20 5:50 AM, Szalay-Bekő Máté wrote:
> > Hi Sebastian,
> >
> > thanks for the more details!
> >
> > One thing I found in your config is that you should use:
> > quorum.auth.learner.saslLoginContext=QuorumLearner
> > quorum.auth.server.saslLoginContext=QuorumServer
> >
> > so instead of loginContext, use saslLoginContext in both lines. I
> found
> > this in the source code, I think the wiki is wrong (I will fix it later).
> > However, actually this didn't really change anything, as the default
> values
> > are anyway
> > QuorumLearner and QuorumServer, so you can even skip these lines from
> the
> > config.
> >
> > I think Rakesh is right, you are seeing exceptions related to not the
> > QuorumSasl, but the ClientSasl. This is why ZooKeeper tries to find the
> > 'Server' section (what is configuring the server during the client-server
> > authentication). The name of this section can be overwritten by the
> > "zookeeper.sasl.serverconfig" system property.
> >
> > Based on the exception, ZooKeeper can not find the 'Server' section in
> > the /opt/zookeeper-cluster/zookeeper/conf/jaas.conf file. Are you sure
> this
> > is the correct jaas.conf? Does the ZooKeeper process have the permissions
> > to open this file? You can specify the jaas config file path for
> ZooKeeper
> > by providing custom system property e.g. by exporting
> > SERVER_JVMFLAGS="-Djava.security.auth.login.config=/path/to/jaas.conf"
> > before starting zkServer.sh
> >
> > Also in the jaas.conf you copied here, you are missing a semicolon from
> the
> > end of the last line in the Server block. I am not sure if it is causing
> > any parsing error, but I always add the semicolon to the end of the last
> > line in the block.
> >
> > Mate
> >
> > On Tue, Feb 11, 2020 at 7:53 PM Sebastian Schmitz <
> > sebastian.schm...@propellerhead.co.nz> wrote:
> >
> >> Hello Rakesh,
> >>
> >> as mentioned in the other mail adding the "Server"to jaas.conf didn't
> help.
> >>
> >> Here are the Configs and Logs (with the Server-part included):
> >>
> >> jaas.conf:
> >> QuorumServer {
> >> org.apache.zookeeper.server.auth.DigestLoginModule required
> >> user_zookeeper="test";
> >> };
> >>
> >> QuorumClient {
> >> org.apache.zookeeper.server.auth.DigestLoginModule required
> >> username="zookeeper"
> >> password="test";
> >> };
> >>
> >> Server {
> >> org.apache.zookeeper.server.auth.DigestLoginModule required
> >> user_zookeeper="test"
> >> };
> >>
> >> Client {
> >> org.apache.zookeeper.server.auth.DigestLoginModule required
> >> username="zookeeper"
> >> password="test";
> >> };
> >>
> >> zoo.cfg:
> >> # The number of milliseconds of each tick
> >> tickTime=2000
> >> # The number of ticks that the initial
> >> # synchronization phase can take
> >> initLimit=10
> >> # The number of ticks that can pass between
> >> # sending a request and getting an
Re: Enabling Auth between Zookeeper Servers
Hi Sebastian,
> But I still get the same exception.
at this point I don't know why this happen... Adding the Server section to
the jaas config should have helped. Unfortunately the exact exception is
not printed out into the logs, just the error message, so it is hard to
find out more details.
I will try to reproduce your case with 3.5.6 locally and see if it works. I
never actually used digest authentication before... we always use kerberos
in production. If it works, I will share my configs / dockerfiles and send
you a patched version with more debug info printed out.
> Why would configuring quorum-auth also enable client-server-auth?
it is not very logical indeed... if I see it right, based on the code once
you set the java.security.auth.login.config property, then ZooKeeper
assumes you want to use server-client sasl authentication. I guess the
quorum-auth feature was added later and they introduced 'enable' config
property for this, but forget to introduce the same config for the client
authentication. I also guess most of the people are interested in the
client authentication and it is rare that someone does't need that but
needs quorum auth. Still, the current behaviour is not good I think. I will
submit a jira ticket requesting an improvement here when I will have time,
but feel free to submit it yourself if you wish.
Kind regards,
Mate
On Thu, Feb 13, 2020 at 7:41 PM Sebastian Schmitz <
sebastian.schm...@propellerhead.co.nz> wrote:
> Hey Mate,
>
> I checked the java.env-file and it contains:
>
>
> SERVER_JVMFLAGS="-Djava.security.auth.login.config=/opt/zookeeper-cluster/zookeeper/conf/jaas.conf"
>
> which is exactly the place where the pasted jaas.conf is placed.
>
> I also just changed the config to be saslLoginContext and added the
> missing semicolon.
>
> But I still get the same exception.
>
> Why would configuring quorum-auth also enable client-server-auth?
>
> Thanks
>
> Sebastian
>
>
> On 13-Feb-20 5:50 AM, Szalay-Bekő Máté wrote:
> > Hi Sebastian,
> >
> > thanks for the more details!
> >
> > One thing I found in your config is that you should use:
> > quorum.auth.learner.saslLoginContext=QuorumLearner
> > quorum.auth.server.saslLoginContext=QuorumServer
> >
> > so instead of loginContext, use saslLoginContext in both lines. I
> found
> > this in the source code, I think the wiki is wrong (I will fix it later).
> > However, actually this didn't really change anything, as the default
> values
> > are anyway
> > QuorumLearner and QuorumServer, so you can even skip these lines from
> the
> > config.
> >
> > I think Rakesh is right, you are seeing exceptions related to not the
> > QuorumSasl, but the ClientSasl. This is why ZooKeeper tries to find the
> > 'Server' section (what is configuring the server during the client-server
> > authentication). The name of this section can be overwritten by the
> > "zookeeper.sasl.serverconfig" system property.
> >
> > Based on the exception, ZooKeeper can not find the 'Server' section in
> > the /opt/zookeeper-cluster/zookeeper/conf/jaas.conf file. Are you sure
> this
> > is the correct jaas.conf? Does the ZooKeeper process have the permissions
> > to open this file? You can specify the jaas config file path for
> ZooKeeper
> > by providing custom system property e.g. by exporting
> > SERVER_JVMFLAGS="-Djava.security.auth.login.config=/path/to/jaas.conf"
> > before starting zkServer.sh
> >
> > Also in the jaas.conf you copied here, you are missing a semicolon from
> the
> > end of the last line in the Server block. I am not sure if it is causing
> > any parsing error, but I always add the semicolon to the end of the last
> > line in the block.
> >
> > Mate
> >
> > On Tue, Feb 11, 2020 at 7:53 PM Sebastian Schmitz <
> > sebastian.schm...@propellerhead.co.nz> wrote:
> >
> >> Hello Rakesh,
> >>
> >> as mentioned in the other mail adding the "Server"to jaas.conf didn't
> help.
> >>
> >> Here are the Configs and Logs (with the Server-part included):
> >>
> >> jaas.conf:
> >> QuorumServer {
> >> org.apache.zookeeper.server.auth.DigestLoginModule required
> >> user_zookeeper="test";
> >> };
> >>
> >> QuorumClient {
> >> org.apache.zookeeper.server.auth.DigestLoginModule required
> >> username="zookeeper"
> >> password="test";
> >> };
> >>
> >> Server {
> >> org.apache.zookeeper.server.auth.DigestLoginModule required
> >> user_zookeeper="test"
> >> };
> >>
> >> Client {
> >> org.apache.zookeeper.server.auth.DigestLoginModule required
> >> username="zookeeper"
> >> password="test";
> >> };
> >>
> >> zoo.cfg:
> >> # The number of milliseconds of each tick
> >> tickTime=2000
> >> # The number of ticks that the initial
> >> # synchronization phase can take
> >> initLimit=10
> >> # The number of ticks that can pass between
> >> # sending a request and getting an acknowledgement
> >> syncLimit=5
> >> # the directory where the snapshot is stored.
> >> # do not use /tmp for
Re: Enabling Auth between Zookeeper Servers
Hey Mate,
I checked the java.env-file and it contains:
SERVER_JVMFLAGS="-Djava.security.auth.login.config=/opt/zookeeper-cluster/zookeeper/conf/jaas.conf"
which is exactly the place where the pasted jaas.conf is placed.
I also just changed the config to be saslLoginContext and added the
missing semicolon.
But I still get the same exception.
Why would configuring quorum-auth also enable client-server-auth?
Thanks
Sebastian
On 13-Feb-20 5:50 AM, Szalay-Bekő Máté wrote:
Hi Sebastian,
thanks for the more details!
One thing I found in your config is that you should use:
quorum.auth.learner.saslLoginContext=QuorumLearner
quorum.auth.server.saslLoginContext=QuorumServer
so instead of loginContext, use saslLoginContext in both lines. I found
this in the source code, I think the wiki is wrong (I will fix it later).
However, actually this didn't really change anything, as the default values
are anyway
QuorumLearner and QuorumServer, so you can even skip these lines from the
config.
I think Rakesh is right, you are seeing exceptions related to not the
QuorumSasl, but the ClientSasl. This is why ZooKeeper tries to find the
'Server' section (what is configuring the server during the client-server
authentication). The name of this section can be overwritten by the
"zookeeper.sasl.serverconfig" system property.
Based on the exception, ZooKeeper can not find the 'Server' section in
the /opt/zookeeper-cluster/zookeeper/conf/jaas.conf file. Are you sure this
is the correct jaas.conf? Does the ZooKeeper process have the permissions
to open this file? You can specify the jaas config file path for ZooKeeper
by providing custom system property e.g. by exporting
SERVER_JVMFLAGS="-Djava.security.auth.login.config=/path/to/jaas.conf"
before starting zkServer.sh
Also in the jaas.conf you copied here, you are missing a semicolon from the
end of the last line in the Server block. I am not sure if it is causing
any parsing error, but I always add the semicolon to the end of the last
line in the block.
Mate
On Tue, Feb 11, 2020 at 7:53 PM Sebastian Schmitz <
sebastian.schm...@propellerhead.co.nz> wrote:
Hello Rakesh,
as mentioned in the other mail adding the "Server"to jaas.conf didn't help.
Here are the Configs and Logs (with the Server-part included):
jaas.conf:
QuorumServer {
org.apache.zookeeper.server.auth.DigestLoginModule required
user_zookeeper="test";
};
QuorumClient {
org.apache.zookeeper.server.auth.DigestLoginModule required
username="zookeeper"
password="test";
};
Server {
org.apache.zookeeper.server.auth.DigestLoginModule required
user_zookeeper="test"
};
Client {
org.apache.zookeeper.server.auth.DigestLoginModule required
username="zookeeper"
password="test";
};
zoo.cfg:
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/mnt/zk_data
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
#
http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1
dataLogDir=/mnt/zk_data_log
autopurge.snapRetainCount=3
autopurge.purgeInterval=24
quorum.auth.enableSasl=true
quorum.auth.learnerRequireSasl=false
quorum.auth.serverRequireSasl=false
quorum.auth.learner.loginContext=QuorumLearner
quorum.auth.server.loginContext=QuorumServer
quorum.cnxn.threads.size=20
authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
secureClientPort=2281
server.1=0.0.0.0:2888:3888
server.2=kafkad02.x.azure.com:2888:3888
server.3=kafkad03.x.azure.com:2888:3888
Server-Log:
Using config: /opt/zookeeper-cluster/zookeeper/bin/../conf/zoo.cfg
Feb 11, 2020 18:43:53 + [1 1] com.newrelic INFO: New Relic Agent:
Loading configuration file "/opt/zookeeper-cluster/newrelic/./newrelic.yml"
Feb 11, 2020 18:43:53 + [1 1] com.newrelic INFO: Using default
collector host: collector.newrelic.com
Feb 11, 2020 18:43:53 + [1 1] com.newrelic INFO: New Relic Agent:
Writing to log file:
/opt/zookeeper-cluster/newrelic/logs/newrelic_agent.log
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by
com.newrelic.weave.weavepackage.NewClassAppender
(file:/opt/zookeeper-cluster/newrelic/newrelic.jar) to method
java.net.URLClassLoader.addURL(java.net.URL)
WARNING:
