Hi All, We are trying to make major version upgrades of Accumulo and Hadoop: Accumulo: 1.8.1 --> 2.0.1 Hadoop: 2.8.2 --> 3.0.3
They run as single non-clustered containers in a Docker environment. We have a separate Zookeeper container running which they talk to at version 3.4.10. Other client containers authenticate with Kerberos in order to retrieve information from Accumulo. We have a KDC running on a separate authentication VM which is reachable by both containers. Once upgrading their major versions we get a problem with their Kerberos authentication. On initialisation, Accumulo and Hadoop run kinit commands to generate themselves ticket-granting-tickets (TGTs) which are valid for 24h, and the full application works as expected. After 24h, however, our client containers can no longer authenticate and access information from Accumulo, despite the clients having valid service tickets for Accumulo. If we manually regenerate the TGT within Accumulo with another kinit command the problem still persists. If we manually change the KDC's configuration to issue 10 minute tickets rather than 24h, then authentication breaks after 10 minutes regardless of each component's krb5.conf file. Pre-upgrade all our containers must have been able to retrieve new tickets once their old ones had expired, but this no longer seems to be the case. The only way to fix the problem is by restarting the containers. Below are the configuration files for the various containers - any variables surrounded by "@" or "<>" are substituted in at runtime and point to valid paths / files / values. Accumulo accumulo.properties: general.kerberos.keytab=@KRB_KEYTAB@<mailto:general.kerberos.keytab=@KRB_KEYTAB@> general.kerberos.principal=@KRB_PRINCIPAL@<mailto:general.kerberos.principal=@KRB_PRINCIPAL@> instance.rpc.sasl.enabled=true instance.secret=@SECRET@<mailto:instance.secret=@SECRET@> instance.security.authenticator=org.apache.accumulo.server.security.handler.KerberosAuthenticator instance.security.authorizor=org.apache.accumulo.server.security.handler.KerberosAuthorizor instance.security.permissionHandler=org.apache.accumulo.server.security.handler.KerberosPermissionHandler instance.volumes=@HDFS_VOLUMES@<mailto:instance.volumes=@HDFS_VOLUMES@> instance.zookeeper.host=@ZOOKEEPERS@<mailto:instance.zookeeper.host=@ZOOKEEPERS@> rpc.sasl.qop=auth trace.token.property.keytab=@KRB_KEYTAB@<mailto:trace.token.property.keytab=@KRB_KEYTAB@> trace.token.type=org.apache.accumulo.core.client.security.tokens.KerberosToken trace.user=@KRB_PRINCIPAL@<mailto:trace.user=@KRB_PRINCIPAL@> tserver.cache.data.size=@CACHE_DATA_SIZE@<mailto:tserver.cache.data.size=@CACHE_DATA_SIZE@> tserver.cache.index.size=@CACHE_INDEX_SIZE@<mailto:tserver.cache.index.size=@CACHE_INDEX_SIZE@> tserver.memory.maps.max=@MEMORY_MAPS_MAX@<mailto:tserver.memory.maps.max=@MEMORY_MAPS_MAX@> tserver.memory.maps.native.enabled=false tserver.sort.buffer.size=@SORT_BUFFER_SIZE@<mailto:tserver.sort.buffer.size=@SORT_BUFFER_SIZE@> tserver.walog.max.size=@WALOG_MAX_SIZE@<mailto:tserver.walog.max.size=@WALOG_MAX_SIZE@> Accumulo accumulo-client.properties: instance.name=accumulo instance.zookeepers=@ZOOKEEPERS@<mailto:instance.zookeepers=@ZOOKEEPERS@> instance.zookeepers.timeout=30s auth.type=kerberos auth.principal=@KRB_PRINCIPAL@<mailto:auth.principal=@KRB_PRINCIPAL@> auth.token=@KRB_KEYTAB@<mailto:auth.token=@KRB_KEYTAB@> sasl.enabled=true sasl.qop=auth sasl.kerberos.server.primary=accumulo Accumulo / Hadoop / Zookeeper krb5.conf (all identical): includedir /etc/krb5.conf.d/ [logging] default = FILE:/var/log/krb5libs.log kdc = FILE:/var/log/krb5kdc.log admin_server = FILE:/var/log/kadmind.log [libdefaults] dns_lookup_realm = false ticket_lifetime = 24h renew_lifetime = 7d forwardable = true rdns = false pkinit_anchors = /etc/pki/tls/certs/ca-bundle.crt default_realm = NRAC.UK #default_ccache_name = KEYRING:persistent:%{uid} udp_preference_limit = 1 [realms] NRAC.UK = { kdc = <ldap-server> admin_server = <ldap-server> default_domain = <our_domain> database_module = openldap_ldapconf } [domain_realm] .<our_domain> = <OUR_DOMAIN> <our_domain> = <OUR_DOMAIN> [dbdefaults] ldap_kerberos_container_dn = cn=krbContainer,dc=nrac,dc=uk [dbmodules] openldap_ldapconf = { db_library = kldap ldap_kdc_dn = "cn=nrac-ldapadm,dc=nrac,dc=uk" ldap_kadmind_dn = "cn=nrac-ldapadm,dc=nrac,dc=uk" ldap_service_password_file = /etc/krb5kdc/service.keyfile ldap_servers = ldaps://<ldap-server> ldap_conns_per_server = 5 } Any help would be much appreciated, many thanks. Alex Sparks Public
