Hi Charles and John, So I made the tickets renewable and regenerated the keytabs for accumulo. The ticket life was set to 1 hour with the renew life set to 1 day. However after the hour is up, I get a different error:
Call to accumulo.test.local/127.0.0.1:8020 failed on local exception: java.io.IOException: java.lang.IllegalStateException: This ticket is no longer valid immediately followed by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] I listed the ticket to make sure I had a valid ticket that was for 1 hour with a renew expiration of 1 day. Then I went in and made sure that the user running the service has a valid ticket, and just in case on a few test added a cron job that renews the ticket before it expires. Either way I get the same error. You mentioned that the system automatically renews the ticket when it expires, and kerberos debug logging enabled I am seeing the following: Found ticket for accumulo/[email protected] to go to krbtgt/[email protected] expiring on Thu Feb 27 07:14:20 PST 2014 Entered Krb5Context.initSecContext with state=STATE_NEW Found ticket for accumulo/[email protected] to go to krbtgt/[email protected] expiring on Thu Feb 27 07:14:20 PST 2014 Found ticket for accumulo/[email protected] to go to hdfs/[email protected] expiring on Thu Feb 27 07:14:20 PST 2014 There is no errors logged for the kerberos ticket creation however the "This ticket is no longer valid " error leads me to believe that the current ticket it had been using was destroyed. Any thoughts? Thanks, Hokie On Wed, Feb 26, 2014 at 3:11 PM, Hyokwon Lee <[email protected]> wrote: > Charles and John, > > Thanks for the help. I am going to make the tickets renewable and give > it a test. I will let you guys know if it works. > > ~Hokie > > > On Wed, Feb 26, 2014 at 2:43 PM, John Vines <[email protected]> wrote: > >> No, they need to be renewable. The system automatically renews them when >> they expire. >> >> >> On Tue, Feb 25, 2014 at 5:08 PM, Hyokwon Lee <[email protected]>wrote: >> >>> Hi Sean, >>> >>> The Kerberos Tickets that are being used are not renewable. Should >>> they be? I assume even if they are after their renewable time expires I >>> will run into the same issue? >>> >>> Thanks, >>> >>> Hokie >>> >>> >>> On Tue, Feb 25, 2014 at 4:39 PM, Sean Busbey >>> <[email protected]>wrote: >>> >>>> Hi Hokie! >>>> >>>> Are the kerberos tickets you're getting renewable? >>>> >>>> -Sean >>>> >>>> >>>> >>>> On Tue, Feb 25, 2014 at 4:35 PM, Hyokwon Lee <[email protected]>wrote: >>>> >>>>> I am currently running into an issue and was hoping someone may have >>>>> some insight to the problem. >>>>> >>>>> Running Accumulo 1.4.3 on top of a Kerberos enabled Hadoop. I followed >>>>> the following instructions in the README: >>>>> >>>>> "If you are running on top of hdfs with kerberos enabled, then you need >>>>> to do >>>>> some extra work. First, create an Accumulo principal >>>>> >>>>> kadmin.local -q "addprinc -randkey accumulo/<host.domain.name>" >>>>> >>>>> where <host.domain.name> is replaced by a fully qualified domain name. >>>>> Export >>>>> the principals to a keytab file. It is safer to create a unique keytab >>>>> file for each >>>>> server, but you can also glob them if you wish. >>>>> >>>>> kadmin.local -q "xst -k accumulo.keytab -glob accumulo*" >>>>> >>>>> Place this file in $ACCUMULO_HOME/conf for every host. It should be owned >>>>> by >>>>> the accumulo user and chmodded to 400. Add the following to the >>>>> accumulo-env.sh >>>>> >>>>> In the accumulo-site.xml file on each node, add settings for >>>>> general.kerberos.keytab >>>>> and general.kerberos.principal, where the keytab setting is the absolute >>>>> path >>>>> to the keytab file ($ACCUMULO_HOME is valid to use) and principal is set >>>>> to >>>>> accumulo/_HOST@<REALM>, where REALM is set to your kerberos realm. You >>>>> may use >>>>> _HOST in lieu of your individual host names. >>>>> >>>>> <property> >>>>> <name>general.kerberos.keytab</name> >>>>> <value>$ACCUMULO_HOME/conf/accumulo.keytab</value> >>>>> </property> >>>>> >>>>> <property> >>>>> <name>general.kerberos.principal</name> >>>>> <value>accumulo/_HOST@MYREALM</value> >>>>> </property> >>>>> >>>>> You can then start up Accumulo as you would with the accumulo user, and >>>>> it will >>>>> automatically handle the kerberos keys needed to access hdfs. >>>>> >>>>> Please Note: You may have issues initializing Accumulo while running >>>>> kerberos HDFS. >>>>> You can resolve this by temporarily granting the accumulo user write >>>>> access to the >>>>> hdfs root directory, running init, and then revoking write permission in >>>>> the root >>>>> directory (be sure to maintain access to the /accumulo directory)." >>>>> >>>>> >>>>> After doing so, got accumulo to come up and initially it states on start >>>>> up that i authenticated using accumulo/[email protected]. >>>>> For the next 24 hour it is happy and everything works. However after >>>>> the 24 hour marker which is when the kerberos ticket expires, I start >>>>> seeing the following errors on all TServers: >>>>> >>>>> >>>>> [securty.UserGroupInformation] ERROR: PrivilegedActionException >>>>> as:accumulo/[email protected] (auth:KERBEROS) >>>>> cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by >>>>> GSSException: No valid credentials provided (Mechanism level: Failed to >>>>> find any Kerberos tgt)] >>>>> >>>>> [ipc.Client] WARN : Exception encountered while connecting to the server >>>>> : javax.security.sasl.SasleEception: GSS initiate failed [Caused by >>>>> GSSException: No valid credentials provided (Mechanism level: Failed to >>>>> find any Kerberos tgt)] >>>>> >>>>> [securty.UserGroupInformation] ERROR: PrivilegedActionException >>>>> as:accumulo/[email protected] (auth:KERBEROS) >>>>> cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by >>>>> GSSException: No valid credentials provided (Mechanism level: Failed to >>>>> find any Kerberos tgt)] >>>>> >>>>> >>>>> And as far as I can tell this just retries and keeps failing. I checked >>>>> the accumulo.keytab file and it is a glob so it has the entries for every >>>>> server that Accumulo is on. Also if I manually do a kinit -kt >>>>> accumulo.keytab accumulo/[email protected] it works find and >>>>> I get a valid ticket. I also made sure everything in hdfs under >>>>> "/accumulo" is owned by accumulo so that doesn't seem to be the problem. >>>>> Also made sure after kiniting I can access the directory path and all sub >>>>> directories. >>>>> >>>>> >>>>> So far the only thing that seems to fix my issue is if I bounce all >>>>> accumulo services and it is happy again. Also until I bounce the >>>>> accumulo services, I get error logs stating it cannot scan any of the >>>>> tables (unable to scan metadata, root_tablet, default_tablet, etc.) Has >>>>> anyone else seen this issue? Did I miss a configuration somewhere >>>>> possibly? >>>>> >>>>> >>>>> Thanks, >>>>> >>>>> >>>>> Hokie >>>>> >>>>> >>>>> >>>> >>> >>> >>> -- >>> __________________________________________ >>> Hyokwon Lee >>> [email protected] >>> >> >> > > > -- > __________________________________________ > Hyokwon Lee > [email protected] > -- __________________________________________ Hyokwon Lee [email protected]
