Hi Charlie, So from my understanding as of Accumulo 1.4.1 the need to manually kinit before starting the process has been resolved. I am currently running 1.4.3 and within the logs it shows that a new ticket is issued at the end of expiration of the former ticket that was being held. What version are you running?
~Hokie On Wed, Feb 26, 2014 at 1:57 PM, Ott, Charlie H. <[email protected]>wrote: > Usually I manually issue the ticket to the accumulo process owners on > all servers in the cluster. Then using a bash script, you can create a > cronjob that will renew the tickets. > > > > Run on master as accumulo user, ensuring passwordless ssh is enabled... > > > > #!/bin/bash > > ssh server2 "kinit -R" > > ssh server3 "kinit -R" > > ssh server3 "kinit -R" > > > > > > So if your tablet servers are running as user "accumulo", then I usually > log in to the 'accumulo' account, perform a kinit, then start-all.sh > > > > I had a lot of issues with renewing the ticket before the expiration date, > so after tweaking the principal properties, eventually I just set the > ticket life for a year on that specific principal. > > > > > > *From:* [email protected][mailto: > [email protected]] *On Behalf > Of *Hyokwon Lee > *Sent:* Wednesday, February 26, 2014 1:40 PM > *To:* [email protected] > *Subject:* Re: Accumulo with Kerberos Error > > > > Hi All, > > > > I have an update to my issue. Have turning on kerberos debug logging, I > have found that a new kerberos ticket is initialized after the current > ticket expires. However table scans fail after the expiration of the > original ticket that was issued at startup of accumulo services. In the > logs I see the following errors: > > > > When trying to scan a table: > > ERROR: java.lang.RuntimeException: > org.apache.accumulo.core.client.impl.AccumuloServerException: Error on > server accumulo.test.local:9997 > > > > Numerous > > !METADATA -> FILE_READ -> /accumulo/tables/!0/root_tablet/xxxx > > Call to accumulo.test.local/127.0.0.1:8020 failed on local exception: > java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed > [Caused by GSSException: No valid credentials provided (Mechanism level: > Failed to find any Kerberos tgt)] > > > > and the log also keeps spamming: > > > > [securty.UserGroupInformation] ERROR: PrivilegedActionException > as:accumulo/[email protected] (auth:KERBEROS) > cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by > GSSException: No valid credentials provided (Mechanism level: Failed to find > any Kerberos tgt)] > > > > Has anyone seen this occur before? I can simply recreate this issue by > letting the accumulo services start up and expire and then just watching > the logs and scanning the tables. > > > > Thanks, > > > > Hokie > > > > On Tue, Feb 25, 2014 at 5:08 PM, Hyokwon Lee <[email protected]> > wrote: > > Hi Sean, > > > > The Kerberos Tickets that are being used are not renewable. Should they > be? I assume even if they are after their renewable time expires I will > run into the same issue? > > > > Thanks, > > > > Hokie > > > > On Tue, Feb 25, 2014 at 4:39 PM, Sean Busbey <[email protected]> > wrote: > > Hi Hokie! > > > > Are the kerberos tickets you're getting renewable? > > > > -Sean > > > > > > On Tue, Feb 25, 2014 at 4:35 PM, Hyokwon Lee <[email protected]> > wrote: > > I am currently running into an issue and was hoping someone may have some > insight to the problem. > > > > Running Accumulo 1.4.3 on top of a Kerberos enabled Hadoop. I followed the > following instructions in the README: > > > > "If you are running on top of hdfs with kerberos enabled, then you need to do > > some extra work. First, create an Accumulo principal > > > > kadmin.local -q "addprinc -randkey accumulo/<host.domain.name>" > > > > where <host.domain.name> is replaced by a fully qualified domain name. Export > > the principals to a keytab file. It is safer to create a unique keytab file > for each > > server, but you can also glob them if you wish. > > > > kadmin.local -q "xst -k accumulo.keytab -glob accumulo*" > > > > Place this file in $ACCUMULO_HOME/conf for every host. It should be owned by > > the accumulo user and chmodded to 400. Add the following to the > accumulo-env.sh > > > > In the accumulo-site.xml file on each node, add settings for > general.kerberos.keytab > > and general.kerberos.principal, where the keytab setting is the absolute path > > to the keytab file ($ACCUMULO_HOME is valid to use) and principal is set to > > accumulo/_HOST@<REALM>, where REALM is set to your kerberos realm. You may use > > _HOST in lieu of your individual host names. > > > > <property> > > <name>general.kerberos.keytab</name> > > <value>$ACCUMULO_HOME/conf/accumulo.keytab</value> > > </property> > > > > <property> > > <name>general.kerberos.principal</name> > > <value>accumulo/_HOST@MYREALM</value> > > </property> > > > > You can then start up Accumulo as you would with the accumulo user, and it > will > > automatically handle the kerberos keys needed to access hdfs. > > > > Please Note: You may have issues initializing Accumulo while running kerberos > HDFS. > > You can resolve this by temporarily granting the accumulo user write access > to the > > hdfs root directory, running init, and then revoking write permission in the > root > > directory (be sure to maintain access to the /accumulo directory)." > > > > > > > > After doing so, got accumulo to come up and initially it states on start up > that i authenticated using accumulo/[email protected]. For the > next 24 hour it is happy and everything works. However after the 24 hour > marker which is when the kerberos ticket expires, I start seeing the > following errors on all TServers: > > > > > > > > [securty.UserGroupInformation] ERROR: PrivilegedActionException > as:accumulo/[email protected] (auth:KERBEROS) > cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by > GSSException: No valid credentials provided (Mechanism level: Failed to find > any Kerberos tgt)] > > [ipc.Client] WARN : Exception encountered while connecting to the server : > javax.security.sasl.SasleEception: GSS initiate failed [Caused by > GSSException: No valid credentials provided (Mechanism level: Failed to find > any Kerberos tgt)] > > [securty.UserGroupInformation] ERROR: PrivilegedActionException > as:accumulo/[email protected] (auth:KERBEROS) > cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by > GSSException: No valid credentials provided (Mechanism level: Failed to find > any Kerberos tgt)] > > > > > > > > > > > > > > > > And as far as I can tell this just retries and keeps failing. I checked the > accumulo.keytab file and it is a glob so it has the entries for every server > that Accumulo is on. Also if I manually do a kinit -kt accumulo.keytab > accumulo/[email protected] it works find and I get a valid > ticket. I also made sure everything in hdfs under "/accumulo" is owned by > accumulo so that doesn't seem to be the problem. Also made sure after > kiniting I can access the directory path and all sub directories. > > > > > > > > > > > > > > > > So far the only thing that seems to fix my issue is if I bounce all accumulo > services and it is happy again. Also until I bounce the accumulo services, I > get error logs stating it cannot scan any of the tables (unable to scan > metadata, root_tablet, default_tablet, etc.) Has anyone else seen this > issue? Did I miss a configuration somewhere possibly? > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > Hokie > > > > > > > > > > > > > > > > > > > > -- > > __________________________________________ > Hyokwon Lee > [email protected] > > > > > > -- > > __________________________________________ > Hyokwon Lee > [email protected] > -- __________________________________________ Hyokwon Lee [email protected]
