Hello all,
I sent this in a couple of months ago, but I'm not sure if I was subscribed to
the mailing list correctly and didn't see any replies - so I thought I'd try
again.
I'm having trouble getting Kerberos authentication to work between Zookeeper
and Accumulo. I am not using any supporting platforms (e.g. Cloudera,
Hortonworks) - this is all being done using Docker with single Accumulo
(1.8.1), Zookeeper (3.4.10) and Hadoop (2.8.2) containers running within their
own Docker network. My KDC is running on a separate CentOS machine, but can be
reached by all of them. I have already managed to integrate Kerberos
authentication with Hadoop and Accumulo, but cannot add Zookeeper client into
the mix.
* The Zookeeper container has this configuration:
/conf/zoo.cfg
# Kerberos Configuration
authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
kerberos.removeHostFromPrincipal=true
kerberos.removeRealmFromPrincipal=true
requireClientAuthScheme=sasl
jaasLoginRenew=3600000
/conf/zookeeper_jaas_server.conf
Server {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
storeKey=true
useTicketCache=false
keyTab="/etc/security/keytabs/zookeeper-server.keytab"
principal="zookeeper/[email protected]<mailto:zookeeper/[email protected]>";
};
export
SERVER_JVMFLAGS="-Djava.security.auth.login.config=/conf/zookeeper_jaas_server.conf
/conf/zookeeper_jaas_client.conf
Client {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
storeKey=true
useTicketCache=false
keyTab="/etc/security/keytabs/zookeeper-client.keytab"
principal="zookeeper-client/[email protected]<mailto:zookeeper-client/[email protected]>";
};
export
CLIENT_JVMFLAGS="-Djava.security.auth.login.config=/conf/zookeeper_jaas_client.conf
* And I've added this configuration to Accumulo:
/usr/local/zookeeper/conf/zookeeper_jaas_client.conf
Client {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
storeKey=true
useTicketCache=false
keyTab="/etc/security/keytabs/accumulo.keytab"
principal="zookeeper-client/[email protected]<mailto:zookeeper-client/[email protected]>";
};
export ACCUMULO_JAAS_CONF="$ZOOKEEPER_HOME/conf/zookeeper_jaas_client.conf"
This means that Zookeeper server starts by authenticating as principle
zookeeper/[email protected]<mailto:zookeeper/[email protected]>
and Accumulo (a Zookeeper client) authenticates with Zookeeper using the
zookeeper-client/[email protected]<mailto:zookeeper-client/[email protected]>
principle.
When starting Accumulo for the first time and running through an
initialisation, this authentication works fine and Accumulo starts as expected.
But when you restart Accumulo, the master fails to start with this error:
2019-02-07 10:52:42,856 [delegation.ZooAuthenticationKeyDistributor] ERROR: Saw
more than one ACL on the node
2019-02-07 10:52:42,858 [delegation.ZooAuthenticationKeyDistributor] ERROR:
Expected /accumulo/ee2ddad6-9df3-43a0-84d4-e9713ae9058c/delegation_token_keys
to have ACLs [31,s{'auth,'} ] but was [31,s{'sasl,'zookeeper-client} ,
31,s{'digest,'accumulo:diZNqb4D71cy0fGxC3meE2ZYWyE=}
]
2019-02-07 10:52:42,858 [master.Master] ERROR: Unexpected exception, exiting
java.lang.IllegalStateException: Delegation token secret key node in ZooKeeper
is not protected.
at
org.apache.accumulo.server.security.delegation.ZooAuthenticationKeyDistributor.initialize(ZooAuthenticationKeyDistributor.java:86)
at org.apache.accumulo.master.Master.run(Master.java:1223)
at org.apache.accumulo.master.Master.main(Master.java:1434)
at
org.apache.accumulo.master.MasterExecutable.execute(MasterExecutable.java:33)
at org.apache.accumulo.start.Main$1.run(Main.java:120)
at java.lang.Thread.run(Thread.java:748)
It looks like the information that Accumulo writes to Zookeeper during the
initialisation has two ACLs associated to it - one for the zookeeper-client
Kerberos principle (sasl) and one for the Accumulo secret (digest). These two
ACLs seem to fail one of the master's start-up checks and causes it to exit.
Is there anyway of disabling the Accumulo secret so that I only have the
Kerberos ACL? Or is there something wrong with the way I've tried to implement
this that would cause this problem?
I'd be very appreciative of any assistance.
Many thanks,
Oliver Jones
