I have been looking at this for 2 days now with no avail... does anyone know
why I would be getting a checksum error when I have validated my keys.
I actually deleted my service principals from kdc DB and added them back with a
human readable password instead of random key. I regenerated my keytab with
those service principal. From namenode, I am able to kinit to the kdc with and
without the keytab. However, when I start the namenode, I still get checksum.
I even tried a different kdc (older 1.8 instead of new 1.9.1) and received the
same exception.
It has to be something simple, but I just can't figure it out.
If anyone has any ideas please let me know.
The latest traces are as follows:
Found key for host/rdcesx10030.race.sas....@obsidian.sas.com(23)
Found key for host/rdcesx10030.race.sas....@obsidian.sas.com(18)
Found ticket for host/rdcesx10030.race.sas....@obsidian.sas.com to go to
krbtgt/obsidian.sas....@obsidian.sas.com expiring on Mon Jul 02 00:33:02 EDT
2012
Entered Krb5Context.acceptSecContext with state=STATE_NEW
Entered Krb5Context.initSecContext with state=STATE_NEW
Found ticket for host/rdcesx10030.race.sas....@obsidian.sas.com to go to
krbtgt/obsidian.sas....@obsidian.sas.com expiring on Mon Jul 02 00:33:02 EDT
2012
Service ticket not found in the subject
>>> Credentials acquireServiceCreds: same realm
Using builtin default etypes for default_tgs_enctypes
default etypes for default_tgs_enctypes: 3 1 23 16 17 18.
>>> EType: sun.security.krb5.internal.crypto.ArcFourHmacEType
>>> CksumType: sun.security.krb5.internal.crypto.RsaMd5CksumType
Checksum failed !
>>> EType: sun.security.krb5.internal.crypto.ArcFourHmacEType
>>> KrbKdcReq send: kdc=cikdc.unx.sas.com UDP:88, timeout=30000, number of
>>> retries =3, #bytes=716
>>> KDCCommunication: kdc=cikdc.unx.sas.com UDP:88, timeout=30000,Attempt =1,
>>> #bytes=716
12/07/01 00:33:05 INFO ipc.Server: IPC Server listener on 8020: readAndProcess
threw exception javax.security.sasl.SaslException: GSS initiate failed [Caused
by GSSException: Failure unspecified at GSS-API level (Mechanism level:
Checksum failed)]. Count of bytes read: 0
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException:
Failure unspecified at GSS-API level (Mechanism level: Checksum failed)]
at
com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:159)
at
org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1007)
at
org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:1180)
at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:537)
at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:344)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: GSSException: Failure unspecified at GSS-API level (Mechanism level:
Checksum failed)
at
sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:741)
at
sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:323)
at
sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:267)
at
com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:137)
... 7 more
Caused by: KrbException: Checksum failed
at
sun.security.krb5.internal.crypto.ArcFourHmacEType.decrypt(ArcFourHmacEType.java:85)
at
sun.security.krb5.internal.crypto.ArcFourHmacEType.decrypt(ArcFourHmacEType.java:77)
at sun.security.krb5.EncryptedData.decrypt(EncryptedData.java:168)
at sun.security.krb5.KrbApReq.authenticate(KrbApReq.java:268)
at sun.security.krb5.KrbApReq.<init>(KrbApReq.java:134)
at
sun.security.jgss.krb5.InitSecContextToken.<init>(InitSecContextToken.java:79)
at
sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:724)
... 10 more
Caused by: java.security.GeneralSecurityException: Checksum failed
at
sun.security.krb5.internal.crypto.dk.ArcFourCrypto.decrypt(ArcFourCrypto.java:388)
at
sun.security.krb5.internal.crypto.ArcFourHmac.decrypt(ArcFourHmac.java:74)
at
sun.security.krb5.internal.crypto.ArcFourHmacEType.decrypt(ArcFourHmacEType.java:83)
... 16 more
Thanks!
_____________________________________________
From: Tony Dean
Sent: Friday, June 29, 2012 4:50 PM
To: 'common-user@hadoop.apache.org'
Subject: hadoop kerberos security / unix kdc
First, I'd like to thank the community for the time and effort they put into
sharing their knowledge...
A few weeks back I was able to configure a secure hadoop/hbase cluster (MIT
1.6.1 Kerberos on cluster) using a Windows Domain Controller/AD for the KDC.
I'm using hadoop 1.0.3 and hbase 0.92.1-security distributions.
Now I am trying setup my own Unix KDC (MIT 1.9.1 Kerberos) against that same
cluster. I know the cluster is configured correctly. The only new piece to
the puzzle is the Unix KDC. The problem occurs when I start the namenode. It
is actually able to login my namenode principal into the KDC just fine. I can
see in the namenode main code that the HTTP Server as well as the RPC server
has been created successfully. It's in the startTrashEmptier() method where
the error occurs. It's like Hadoop is acting as a client and connecting back
into itself (hdfs service) when it receives a checksum error:
12/06/29 15:56:13 INFO security.UserGroupInformation: Login successful for user
host/rdcesx10030.race.sas....@obsidian.sas.com using keytab file
/etc/krb5.keytab
12/06/29 15:56:13 INFO ipc.Server: IPC Server Responder: starting
12/06/29 15:56:13 INFO ipc.Server: IPC Server listener on 8020: starting
Found key for host/rdcesx10030.race.sas....@obsidian.sas.com(18)
Found key for host/rdcesx10030.race.sas....@obsidian.sas.com(3)
Found key for host/rdcesx10030.race.sas....@obsidian.sas.com(16)
Found key for host/rdcesx10030.race.sas....@obsidian.sas.com(17)
Found key for host/rdcesx10030.race.sas....@obsidian.sas.com(23)
Entered Krb5Context.acceptSecContext with state=STATE_NEW
>>> EType: sun.security.krb5.internal.crypto.ArcFourHmacEType
Checksum failed !
12/06/29 15:56:13 INFO ipc.Server: IPC Server listener on 8020: readAndProcess
threw exception javax.security.sasl.SaslException: GSS initiate failed [Caused
by GSSException: Failure unspecified at GSS-API level (Mechanism level:
Checksum failed)]. Count of bytes read: 0
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException:
Failure unspecified at GSS-API level (Mechanism level: Checksum failed)]
at
com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:159)
at
org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1007)
at
org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:1180)
at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:537)
at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:344)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: GSSException: Failure unspecified at GSS-API level (Mechanism level:
Checksum failed)
at
sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:741)
at
sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:323)
at
sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:267)
at
com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:137)
... 7 more
Caused by: KrbException: Checksum failed
at
sun.security.krb5.internal.crypto.ArcFourHmacEType.decrypt(ArcFourHmacEType.java:85)
at
sun.security.krb5.internal.crypto.ArcFourHmacEType.decrypt(ArcFourHmacEType.java:77)
at sun.security.krb5.EncryptedData.decrypt(EncryptedData.java:168)
at sun.security.krb5.KrbApReq.authenticate(KrbApReq.java:268)
at sun.security.krb5.KrbApReq.<init>(KrbApReq.java:134)
at
sun.security.jgss.krb5.InitSecContextToken.<init>(InitSecContextToken.java:79)
at
sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:724)
... 10 more
Caused by: java.security.GeneralSecurityException: Checksum failed
at
sun.security.krb5.internal.crypto.dk.ArcFourCrypto.decrypt(ArcFourCrypto.java:388)
at
sun.security.krb5.internal.crypto.ArcFourHmac.decrypt(ArcFourHmac.java:74)
at
sun.security.krb5.internal.crypto.ArcFourHmacEType.decrypt(ArcFourHmacEType.java:83)
... 16 more
I think it has something to do with the keys in my keytab. Although, I can
kinit into the KDC with all of the principals in my keytab so I don't know what
the problem is.
I read something (not validated though) that there may be some incompatibility
with Hadoop security and MIT 1.9.1.
Any insight here would be greatly appreciated.
Thanks.
Tony Dean
SAS Institute Inc.
Senior Software Developer
919-531-6704
<< OLE Object: Picture (Device Independent Bitmap) >>
