Hey Mate,
now it gets really weird. I get the file not found exception:
'.20-02-16 18:27:50,530 [myid:1] - ERROR
[main:ServerCnxnFactory@246] -
No JAAS configuration section named 'Server' was found in
'/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
java.lang.SecurityException: java.io.IOException:
/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
(No such file or directory)
at
java.base/sun.security.provider.ConfigFile$Spi.<init>(Unknown Source)
at java.base/sun.security.provider.ConfigFile.<init>(Unknown
Source)
at
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown
Source)
at
java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown
Source)
at
java.base/java.lang.reflect.Constructor.newInstance(Unknown
Source)
at java.base/java.lang.Class.newInstance(Unknown Source)
at
java.base/javax.security.auth.login.Configuration$2.run(Unknown
Source)
at
java.base/javax.security.auth.login.Configuration$2.run(Unknown
Source)
at
java.base/java.security.AccessController.doPrivileged(Native
Method)
at
java.base/javax.security.auth.login.Configuration.getConfiguration(Unknown
Source)
at
org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:210)
at
org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:646)
at
org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:143)
at
org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:106)
at
org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:64)
at
org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:128)
at
org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82)
Caused by: java.io.IOException:
/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
(No such file or directory)
at
java.base/sun.security.provider.ConfigFile$Spi.ioException(Unknown
Source)
at
java.base/sun.security.provider.ConfigFile$Spi.init(Unknown
Source)
... 18 more
2020-02-16 18:27:50,566 [myid:1] - ERROR
[main:ZooKeeperServerMain@83] -
Unexpected exception, exiting abnormally
java.io.IOException: No JAAS configuration section named 'Server' was
found in '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
'.
at
org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:247)
at
org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:646)
at
org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:143)
at
org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:106)
at
org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:64)
at
org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:128)
at
org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82)
So I checked the image:
root@2eeeb625500c:/opt/zookeeper-cluster/zookeeper/conf# ls -la
total 32
drwxr-xr-x 2 1010 1011 4096 Feb 16 18:27 .
drwxr-xr-x 12 root root 4096 Feb 16 18:27 ..
-rw-r--r-- 1 1010 1011 535 Jan 30 12:18 configuration.xsl
-rw-r--r-- 1 root root 600 Feb 13 18:32 jaas.conf
-rw-r--r-- 1 root root 101 Feb 11 00:05 java.env
-rw-r--r-- 1 1010 1011 2712 Feb 14 05:49 log4j.properties
-rw-r--r-- 1 root root 1255 Feb 16 18:27 zoo.cfg
-rw-r--r-- 1 1010 1011 922 Feb 14 05:49 zoo_sample.cfg
And tried to output the file it states in the error:
root@2eeeb625500c:/opt/zookeeper-cluster/zookeeper/conf# cat
/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
QuorumServer {
org.apache.zookeeper.server.auth.DigestLoginModule required
user_zookeeper="test";
};
QuorumClient {
org.apache.zookeeper.server.auth.DigestLoginModule required
username="zookeeper"
password="test";
};
Server {
org.apache.zookeeper.server.auth.DigestLoginModule required
user_zookeeper="test";
};
Client {
org.apache.zookeeper.server.auth.DigestLoginModule required
username="zookeeper"
password="test";
};
The weird part now is that the access is set exactly the same as the
zoo.cfg which it can read without problems.
Also changing the access to 666 doesn't change anything. And using
your
config doesn't help either:
jaas.conf:
QuorumServer {
org.apache.zookeeper.server.auth.DigestLoginModule required
user_zookeeper="test";
};
QuorumLearner {
org.apache.zookeeper.server.auth.DigestLoginModule required
username="zookeeper"
password="test";
};
Server {
org.apache.zookeeper.server.auth.DigestLoginModule required
user_zookeeper="test";
};
zoo.cfg:
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/mnt/zk_data
clientPort=2181
standaloneEnabled=true
admin.enableServer=true
localSessionsEnabled=true
localSessionsUpgradingEnabled=true
4lw.commands.whitelist=stat, ruok, conf, isro, wchc, wchp, srvr,
mntr, cons
clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory
authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
quorum.auth.enableSasl=true
quorum.auth.learnerRequireSasl=false
quorum.auth.serverRequireSasl=false
quorum.auth.learner.saslLoginContext=QuorumLearner
quorum.auth.server.saslLoginContext=QuorumServer
dataLogDir=/mnt/zk_data_log
autopurge.snapRetainCount=3
autopurge.purgeInterval=24
quorum.cnxn.threads.size=20
server.1=0.0.0.0:2888:3888
I have no idea what's different now. I'll try to run the stuff from
your
repo and see if that works.
Best regards
Sebastian
On 14-Feb-20 8:11 PM, Szalay-Bekő Máté wrote:
Hi Sebastian!
I was able to setup digest authentication, uploaded my results here:
https://github.com/symat/zookeeper-docker-test
You can see my docker compose file:
https://github.com/symat/zookeeper-docker-test/blob/master/3_nodes_digest_quorum_auth.yml
also the zoo.cfg template:
https://github.com/symat/zookeeper-docker-test/blob/master/conf/digest_zoo.cfg
and the jaas.cfg file:
https://github.com/symat/zookeeper-docker-test/blob/master/conf/digest_jaas.conf
It works for me, using ZooKeeper 3.5.6. Although I haven't follow
your
config everywhere.
Still, I wasn't able to reproduce your exception, only when I
actually
deleted the jaas config file. Are you sure that the ZooKeeper
process in
docker can see / open that file?
I created a patched ZooKeeper 3.5.6 for you (you can download from
here:
https://drive.google.com/open?id=1KEPjNkiKf937jMJHAicwW9WATEuyRZIo),
where
more details are printed in case of errors. E.g. in my case when I
deleted
the jaas config file, I get:
zoo1_1 | 2020-02-14 07:04:33,288 [myid:1] - ERROR
[main:ServerCnxnFactory@246] - No JAAS configuration section named
'Server'
was found in '/scripts/conf/digest_jaas.conf'.
zoo1_1 | java.lang.SecurityException: java.io.IOException:
/scripts/conf/digest_jaas.conf (No such file or directory)
zoo1_1 | at
sun.security.provider.ConfigFile$Spi.<init>(ConfigFile.java:137)
zoo1_1 | at
sun.security.provider.ConfigFile.<init>(ConfigFile.java:102)
zoo1_1 | at
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
zoo1_1 | at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
zoo1_1 | at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
zoo1_1 | at
java.lang.reflect.Constructor.newInstance(Constructor.java:423)
zoo1_1 | at java.lang.Class.newInstance(Class.java:442)
zoo1_1 | at
javax.security.auth.login.Configuration$2.run(Configuration.java:255)
zoo1_1 | at
javax.security.auth.login.Configuration$2.run(Configuration.java:247)
zoo1_1 | at java.security.AccessController.doPrivileged(Native
Method)
zoo1_1 | at
javax.security.auth.login.Configuration.getConfiguration(Configuration.java:246)
zoo1_1 | at
org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:210)
zoo1_1 | at
org.apache.zookeeper.server.NettyServerCnxnFactory.configure(NettyServerCnxnFactory.java:383)
zoo1_1 | at
org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:148)
zoo1_1 | at
org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:123)
zoo1_1 | at
org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82)
zoo1_1 | Caused by: java.io.IOException:
/scripts/conf/digest_jaas.conf
(No such file or directory)
zoo1_1 | at
sun.security.provider.ConfigFile$Spi.ioException(ConfigFile.java:666)
zoo1_1 | at
sun.security.provider.ConfigFile$Spi.init(ConfigFile.java:262)
zoo1_1 | at
sun.security.provider.ConfigFile$Spi.<init>(ConfigFile.java:135)
zoo1_1 | ... 15 more
z
Kind regards,
Mate
On Fri, Feb 14, 2020 at 7:12 AM sagar shukla
<sa_shu...@yahoo.com.invalid
wrote:
O
Sent from Yahoo Mail on Android
On Fri, Feb 14, 2020 at 11:02 AM, Szalay-Bekő Máté<
szalay.beko.m...@gmail.com> wrote: Hi Sebastian,
But I still get the same exception.
at this point I don't know why this happen... Adding the Server
section
to
the jaas config should have helped. Unfortunately the exact
exception is
not printed out into the logs, just the error message, so it is
hard to
find out more details.
I will try to reproduce your case with 3.5.6 locally and see if it
works. I
never actually used digest authentication before... we always use
kerberos
in production. If it works, I will share my configs / dockerfiles
and
send
you a patched version with more debug info printed out.
Why would configuring quorum-auth also enable client-server-auth?
it is not very logical indeed... if I see it right, based on the
code
once
you set the java.security.auth.login.config property, then ZooKeeper
assumes you want to use server-client sasl authentication. I
guess the
quorum-auth feature was added later and they introduced 'enable'
config
property for this, but forget to introduce the same config for the
client
authentication. I also guess most of the people are interested in
the
client authentication and it is rare that someone does't need
that but
needs quorum auth. Still, the current behaviour is not good I
think. I
will
submit a jira ticket requesting an improvement here when I will have
time,
but feel free to submit it yourself if you wish.
Kind regards,
Mate
On Thu, Feb 13, 2020 at 7:41 PM Sebastian Schmitz <
sebastian.schm...@propellerhead.co.nz> wrote:
Hey Mate,
I checked the java.env-file and it contains:
SERVER_JVMFLAGS="-Djava.security.auth.login.config=/opt/zookeeper-cluster/zookeeper/conf/jaas.conf"
which is exactly the place where the pasted jaas.conf is placed.
I also just changed the config to be saslLoginContext and added the
missing semicolon.
But I still get the same exception.
Why would configuring quorum-auth also enable client-server-auth?
Thanks
Sebastian
On 13-Feb-20 5:50 AM, Szalay-Bekő Máté wrote:
Hi Sebastian,
thanks for the more details!
One thing I found in your config is that you should use:
quorum.auth.learner.saslLoginContext=QuorumLearner
quorum.auth.server.saslLoginContext=QuorumServer
so instead of loginContext, use saslLoginContext in both
lines. I
found
this in the source code, I think the wiki is wrong (I will fix it
later).
However, actually this didn't really change anything, as the
default
values
are anyway
QuorumLearner and QuorumServer, so you can even skip these
lines from
the
config.
I think Rakesh is right, you are seeing exceptions related to
not the
QuorumSasl, but the ClientSasl. This is why ZooKeeper tries to
find
the
'Server' section (what is configuring the server during the
client-server
authentication). The name of this section can be overwritten by
the
"zookeeper.sasl.serverconfig" system property.
Based on the exception, ZooKeeper can not find the 'Server'
section in
the /opt/zookeeper-cluster/zookeeper/conf/jaas.conf file. Are
you sure
this
is the correct jaas.conf? Does the ZooKeeper process have the
permissions
to open this file? You can specify the jaas config file path for
ZooKeeper
by providing custom system property e.g. by exporting
SERVER_JVMFLAGS="-Djava.security.auth.login.config=/path/to/jaas.conf"
before starting zkServer.sh
Also in the jaas.conf you copied here, you are missing a semicolon
from
the
end of the last line in the Server block. I am not sure if it is
causing
any parsing error, but I always add the semicolon to the end of
the
last
line in the block.
Mate
On Tue, Feb 11, 2020 at 7:53 PM Sebastian Schmitz <
sebastian.schm...@propellerhead.co.nz> wrote:
Hello Rakesh,
as mentioned in the other mail adding the "Server"to jaas.conf
didn't
help.
Here are the Configs and Logs (with the Server-part included):
jaas.conf:
QuorumServer {
org.apache.zookeeper.server.auth.DigestLoginModule required
user_zookeeper="test";
};
QuorumClient {
org.apache.zookeeper.server.auth.DigestLoginModule required
username="zookeeper"
password="test";
};
Server {
org.apache.zookeeper.server.auth.DigestLoginModule required
user_zookeeper="test"
};
Client {
org.apache.zookeeper.server.auth.DigestLoginModule required
username="zookeeper"
password="test";
};
zoo.cfg:
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/mnt/zk_data
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
#
http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1
dataLogDir=/mnt/zk_data_log
autopurge.snapRetainCount=3
autopurge.purgeInterval=24
quorum.auth.enableSasl=true
quorum.auth.learnerRequireSasl=false
quorum.auth.serverRequireSasl=false
quorum.auth.learner.loginContext=QuorumLearner
quorum.auth.server.loginContext=QuorumServer
quorum.cnxn.threads.size=20
authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
secureClientPort=2281
server.1=0.0.0.0:2888:3888
server.2=kafkad02.x.azure.com:2888:3888
server.3=kafkad03.x.azure.com:2888:3888
Server-Log:
Using config:
/opt/zookeeper-cluster/zookeeper/bin/../conf/zoo.cfg
Feb 11, 2020 18:43:53 +0000 [1 1] com.newrelic INFO: New Relic
Agent:
Loading configuration file
"/opt/zookeeper-cluster/newrelic/./newrelic.yml"
Feb 11, 2020 18:43:53 +0000 [1 1] com.newrelic INFO: Using
default
collector host: collector.newrelic.com
Feb 11, 2020 18:43:53 +0000 [1 1] com.newrelic INFO: New Relic
Agent:
Writing to log file:
/opt/zookeeper-cluster/newrelic/logs/newrelic_agent.log
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by
com.newrelic.weave.weavepackage.NewClassAppender
(file:/opt/zookeeper-cluster/newrelic/newrelic.jar) to method
java.net.URLClassLoader.addURL(java.net.URL)
WARNING: Please consider reporting this to the maintainers of
com.newrelic.weave.weavepackage.NewClassAppender
WARNING: Use --illegal-access=warn to enable warnings of further
illegal
reflective access operations
WARNING: All illegal access operations will be denied in a future
release
2020-02-11 18:43:59,257 [myid:] - INFO
[main:QuorumPeerConfig@136] -
Reading configuration from:
/opt/zookeeper-cluster/zookeeper/bin/../conf/zoo.cfg
2020-02-11 18:43:59,477 [myid:] - INFO
[main:QuorumPeer$QuorumServer@185] - Resolved hostname:
kafkad02.x.azure.com to address: kafkad02.x.azure.com/1.2.3.4
2020-02-11 18:43:59,477 [myid:] - INFO
[main:QuorumPeer$QuorumServer@185] - Resolved hostname:
0.0.0.0 to
address: /0.0.0.0
2020-02-11 18:43:59,666 [myid:] - INFO
[main:QuorumPeer$QuorumServer@185] - Resolved hostname:
kafkad03.x.azure.com to address: kafkad03.x.azure.com/1.2.3.5
2020-02-11 18:43:59,666 [myid:] - INFO
[main:QuorumPeerConfig@398] -
Defaulting to majority quorums
2020-02-11 18:43:59,677 [myid:1] - INFO
[main:DatadirCleanupManager@78
]
- autopurge.snapRetainCount set to 3
2020-02-11 18:43:59,677 [myid:1] - INFO
[main:DatadirCleanupManager@79
]
- autopurge.purgeInterval set to 24
2020-02-11 18:43:59,732 [myid:1] - INFO
[PurgeTask:DatadirCleanupManager$PurgeTask@138] - Purge task
started.
2020-02-11 18:43:59,749 [myid:1] - INFO
[main:QuorumPeerMain@130] -
Starting quorum peer
2020-02-11 18:43:59,788 [myid:1] - INFO
[main:ServerCnxnFactory@117]
-
Using org.apache.zookeeper.server.NIOServerCnxnFactory as server
connection factory
2020-02-11 18:43:59,804 [myid:1] - INFO
[PurgeTask:DatadirCleanupManager$PurgeTask@144] - Purge task
completed.
'.20-02-11 18:43:59,826 [myid:1] - ERROR
[main:ServerCnxnFactory@210
]
-
No JAAS configuration section named 'Server' was foundin
'/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
2020-02-11 18:43:59,827 [myid:1] - ERROR
[main:QuorumPeerMain@92] -
Unexpected exception, exiting abnormally
java.io.IOException: No JAAS configuration section named
'Server' was
foundin '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
'.
at
org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:211)
at
org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:82)
at
org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:133)
at
org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:114)
at
org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:81)
Best regards
Sebastian
On 12-Feb-20 2:36 AM, Rakesh Radhakrishnan wrote:
java.io.IOException: No JAAS configuration section named
'Server'
I could see you have enabled client-server authentication as
well.
It
looks to me that the error is coming from that. Please share the
complete error logs to trace it.
Have you configured "*Server*" section along with the
"*QuorumServer*"
and "*QuorumClient*" sections? If not, please configure
"*Server*"
section along with others and try it out.
Reference:
https://cwiki.apache.org/confluence/display/ZOOKEEPER/Client-Server+mutual+authentication
image.png
Thanks,
Rakesh
On Tue, Feb 11, 2020 at 7:26 AM Sebastian Schmitz
<sebastian.schm...@propellerhead.co.nz
<mailto:sebastian.schm...@propellerhead.co.nz>> wrote:
Hello,
I'm currently looking into enabling the Auth between
Zookeeper-Servers
and found this documentation:
https://cwiki.apache.org/confluence/display/ZOOKEEPER/Server-Server+mutual+authentication
However, when I use the config from the document (for
Digest-MD5)
I get
this exception in Zookeeper 3.4.14 and also 3.5.6,
which I
tried
because
I thought using latest version could help:
java.io.IOException: No JAAS configuration section named
'Server'
was
found in '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
And of course that's right, because there's only
QuorumServer
and
QuorumClient in the jaas.conf:
jaas.conf:
QuorumServer {
org.apache.zookeeper.server.auth.DigestLoginModule
required
user_zookeeper="test";
};
QuorumClient {
org.apache.zookeeper.server.auth.DigestLoginModule
required
username="zookeeper"
password="test";
};
I also tried renaming the QuorumServer to just
"Server". No
change.
My zoo.cfg:
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/mnt/zk_data
clientPort=2181
dataLogDir=/mnt/zk_data_log
autopurge.snapRetainCount=3
autopurge.purgeInterval=24
quorum.auth.enableSasl=true
quorum.auth.learnerRequireSasl=false
quorum.auth.serverRequireSasl=false
quorum.auth.learner.loginContext=QuorumLearner
quorum.auth.server.loginContext=QuorumServer
quorum.cnxn.threads.size=20
authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
�� secureClientPort=2281
server.1=0.0.0.0:2888:3888
Any idea what I could try? Or maybe there's some better
document
on how
to achieve this?
Thank you
Sebastian
--
DISCLAIMER
This email contains information that is confidential
and which
may be
legally privileged. If you have received this email in
error
please
notify the sender immediately and delete the email.
This email is intended
solely for the use of the intended recipient and you
may not
use
or
disclose this email in any way.
--
DISCLAIMER
This email contains information that is confidential and which
may be
legally privileged. If you have received this email in error
please
notify the sender immediately and delete the email.
This email is intended
solely for the use of the intended recipient and you may not
use or
disclose this email in any way.
--
DISCLAIMER
This email contains information that is confidential and which
may be
legally privileged. If you have received this email in error please
notify the sender immediately and delete the email.
This email is intended
solely for the use of the intended recipient and you may not use or
disclose this email in any way.
--
DISCLAIMER
This email contains information that is confidential and which
may be
legally privileged. If you have received this email in error please
notify the sender immediately and delete the email.
This email is intended
solely for the use of the intended recipient and you may not use or
disclose this email in any way.
