[
https://issues.apache.org/jira/browse/HADOOP-14441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16019838#comment-16019838
]
Wei-Chiu Chuang commented on HADOOP-14441:
------------------------------------------
Here's some sample stack trace. Note I added additional debug logs so it's
quite cluttered.
2017-05-17 18:44:53,382 DEBUG LoadBalancingKMSClientProvider - trying provider
https://weichiu-foo-3.example.com:16000/kms/v1/
2017-05-17 18:44:53,383 WARN Token - Cannot find class for token kind kms-dt
2017-05-17 18:44:53,384 WARN Token - Cannot find class for token kind kms-dt
2017-05-17 18:44:53,384 DEBUG KMSClientProvider - KMS provider
[https://weichiu-foo-3.example.com:16000/kms/v1/] actual ugi = foo
(auth:KERBEROS) subject=Subject:
Principal: UnixPrincipal: foo
Principal: UnixNumericUserPrincipal: 2004
Principal: UnixNumericGroupPrincipal [Primary Group]: 2004
Principal: foo
Private Credential: tokenMap: key=172.31.117.206:8032 value=Kind:
RM_DELEGATION_TOKEN, Service: 172.31.117.206:8032, Ident: 00 18 61 74 74 69 76
69 6f 40 47 43 45 2e 43 4c 4f
55 44 45 52 41 2e 43 4f 4d 04 79 61 72 6e 00 8a 01 5c 19 39 a4 55 8a 01 5c 3d
46 28 55 1a 02;
key=ha-hdfs:ns1 value=Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:ns1, Ident:
(HDFS_DELEGATION_TOKEN token 110 for foo);
key=172.31.123.173:16000 value=Kind: kms-dt, Service: 172.31.123.173:16000,
Ident: 00 07 61 74 74 69 76 69 6f 04 79 61 72 6e 00 8a 01 5c 19 39 a4 43 8a 01
5c 3d 46 28 43 25 22;
secretKeysMap:
current ugi=foo (auth:KERBEROS) subject=Subject:
Principal: UnixPrincipal: foo
Principal: UnixNumericUserPrincipal: 2004
Principal: UnixNumericGroupPrincipal [Primary Group]: 2004
Principal: foo
Private Credential: tokenMap: key=172.31.117.206:8032 value=Kind:
RM_DELEGATION_TOKEN, Service: 172.31.117.206:8032, Ident: 00 18 61 74 74 69 76
69 6f 40 47 43 45 2e 43 4c 4f 55 44 45 52 41 2e 43 4f 4d 04 79 61 72 6e 00 8a
01 5c 19 39 a4 55 8a 01 5c 3d 46 28 55 1a 02;
key=ha-hdfs:ns1 value=Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:ns1, Ident:
(HDFS_DELEGATION_TOKEN token 110 for foo);
key=172.31.123.173:16000 value=Kind: kms-dt, Service: 172.31.123.173:16000,
Ident: 00 07 61 74 74 69 76 69 6f 04 79 61 72 6e 00 8a 01 5c 19 39 a4 43 8a 01
5c 3d 46 28 43 25 22;
secretKeysMap:
url=https://weichiu-foo-3.example.com:16000/kms/v1/keyversion/yH32H7e2tnhd38HGrb45OlrG4xHYJheOs4ITA5NhZbr/_eek?eek_op=decrypt
authToken=null doAsUser=null
2017-05-17 18:44:53,385 WARN Token - Cannot find class for token kind kms-dt
2017-05-17 18:44:53,386 WARN Token - Cannot find class for token kind kms-dt
2017-05-17 18:44:53,388 DEBUG UserGroupInformation - PrivilegedAction as:foo
(auth:KERBEROS) subject=Subject:
Principal: UnixPrincipal: foo
Principal: UnixNumericUserPrincipal: 2004
Principal: UnixNumericGroupPrincipal [Primary Group]: 2004
Principal: foo
Private Credential: tokenMap: key=172.31.117.206:8032 value=Kind:
RM_DELEGATION_TOKEN, Service: 172.31.117.206:8032, Ident: 00 18 61 74 74 69 76
69 6f 40 47 43 45 2e 43 4c 4f 55 44 45 52 41 2e 43 4f 4d 04 79 61 72 6e 00 8a
01 5c 19 39 a4 55 8a 01 5c 3d 46 28 55 1a 02;
key=ha-hdfs:ns1 value=Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:ns1, Ident:
(HDFS_DELEGATION_TOKEN token 110 for foo);
key=172.31.123.173:16000 value=Kind: kms-dt, Service: 172.31.123.173:16000,
Ident: 00 07 61 74 74 69 76 69 6f 04 79 61 72 6e 00 8a 01 5c 19 39 a4 43 8a 01
5c 3d 46 28 43 25 22;
secretKeysMap:
from:org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:489).
subject=Subject:
Principal: UnixPrincipal: foo
Principal: UnixNumericUserPrincipal: 2004
Principal: UnixNumericGroupPrincipal [Primary Group]: 2004
Principal: foo
Private Credential: tokenMap: key=172.31.117.206:8032 value=Kind:
RM_DELEGATION_TOKEN, Service: 172.31.117.206:8032, Ident: 00 18 61 74 74 69 76
69 6f 40 47 43 45 2e 43 4c 4f 55 44 45 52 41 2e 43 4f 4d 04 79 61 72 6e 00 8a
01 5c 19 39 a4 55 8a 01 5c 3d 46 28 55 1a 02;
key=ha-hdfs:ns1 value=Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:ns1, Ident:
(HDFS_DELEGATION_TOKEN token 110 for foo);
key=172.31.123.173:16000 value=Kind: kms-dt, Service: 172.31.123.173:16000,
Ident: 00 07 61 74 74 69 76 69 6f 04 79 61 72 6e 00 8a 01 5c 19 39 a4 43 8a 01
5c 3d 46 28 43 25 22;
secretKeysMap:
.java.lang.Throwable
at
org.apache.hadoop.security.UserGroupInformation.logPrivilegedAction(UserGroupInformation.java:1687)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1662)
at
org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:489)
at
org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:787)
at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$3.call(LoadBalancingKMSClientProvider.java:192)
at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$3.call(LoadBalancingKMSClientProvider.java:188)
at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:97)
at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.decryptEncryptedKey(LoadBalancingKMSClientProvider.java:188)
at
org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:388)
at
org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:1381)
at
org.apache.hadoop.hdfs.DFSClient.createWrappedInputStream(DFSClient.java:1451)
at
org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:305)
at
org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:299)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:312)
2017-05-17 18:44:53,388 DEBUG DelegationTokenAuthenticatedURL - token is not set
2017-05-17 18:44:53,389 DEBUG Credentials - addAll: called by
java.lang.Throwable
at org.apache.hadoop.security.Credentials.addAll(Credentials.java:315)
at org.apache.hadoop.security.Credentials.addAll(Credentials.java:302)
at org.apache.hadoop.security.Credentials.<init>(Credentials.java:77)
at
org.apache.hadoop.security.UserGroupInformation.getCredentials(UserGroupInformation.java:1480)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.openConnection(DelegationTokenAuthenticatedURL.java:294)
at
org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:494)
at
org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:489)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1663)
at
org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:489)
at
org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:787)
at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$3.call(LoadBalancingKMSClientProvider.java:192)
at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$3.call(LoadBalancingKMSClientProvider.java:188)
at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:97)
at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.decryptEncryptedKey(LoadBalancingKMSClientProvider.java:188)
at
org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:388)
at
org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:1381)
at
org.apache.hadoop.hdfs.DFSClient.createWrappedInputStream(DFSClient.java:1451)
at
org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:305)
at
org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:299)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:312)
2017-05-17 18:44:53,390 DEBUG Credentials - addAll token key
172.31.117.206:8032 to this=tokenMap: secretKeysMap: : Kind:
RM_DELEGATION_TOKEN, Service: 172.31.117.206:8032, Ident: 00 18 61 74 74 69 76
69 6f 40 47 43 45 2e 43 4c 4f 55 44 45 52 41 2e 43 4f 4d 04 79 61 72 6e 00 8a
01 5c 19 39 a4 55 8a 01 5c 3d 46 28 55 1a 02
2017-05-17 18:44:53,390 DEBUG Credentials - addAll token key ha-hdfs:ns1 to
this=tokenMap: key=172.31.117.206:8032 value=Kind: RM_DELEGATION_TOKEN,
Service: 172.31.117.206:8032, Ident: 00 18 61 74 74 69 76 69 6f 40 47 43 45 2e
43 4c 4f 55 44 45 52 41 2e 43 4f 4d 04 79 61 72 6e 00 8a 01 5c 19 39 a4 55 8a
01 5c 3d 46 28 55 1a 02;
secretKeysMap: : Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:ns1, Ident:
(HDFS_DELEGATION_TOKEN token 110 for foo)
2017-05-17 18:44:53,391 WARN Token - Cannot find class for token kind kms-dt
2017-05-17 18:44:53,391 DEBUG Credentials - addAll token key
172.31.123.173:16000 to this=tokenMap: key=172.31.117.206:8032 value=Kind:
RM_DELEGATION_TOKEN, Service: 172.31.117.206:8032, Ident: 00 18 61 74 74 69 76
69 6f 40 47 43 45 2e 43 4c 4f 55 44 45 52 41 2e 43 4f 4d 04 79 61 72 6e 00 8a
01 5c 19 39 a4 55 8a 01 5c 3d 46 28 55 1a 02;
key=ha-hdfs:ns1 value=Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:ns1, Ident:
(HDFS_DELEGATION_TOKEN token 110 for foo);
secretKeysMap: : Kind: kms-dt, Service: 172.31.123.173:16000, Ident: 00 07 61
74 74 69 76 69 6f 04 79 61 72 6e 00 8a 01 5c 19 39 a4 43 8a 01 5c 3d 46 28 43
25 22
2017-05-17 18:44:53,392 WARN Token - Cannot find class for token kind kms-dt
2017-05-17 18:44:53,392 DEBUG DelegationTokenAuthenticatedURL - credentials:
tokenMap: key=172.31.117.206:8032 value=Kind: RM_DELEGATION_TOKEN, Service:
172.31.117.206:8032, Ident: 00
18 61 74 74 69 76 69 6f 40 47 43 45 2e 43 4c 4f 55 44 45 52 41 2e 43 4f 4d 04
79 61 72 6e 00 8a 01 5c 19 39 a4 55 8a 01 5c 3d 46 28 55 1a 02;
key=ha-hdfs:ns1 value=Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:ns1, Ident:
(HDFS_DELEGATION_TOKEN token 110 for foo);
key=172.31.123.173:16000 value=Kind: kms-dt, Service: 172.31.123.173:16000,
Ident: 00 07 61 74 74 69 76 69 6f 04 79 61 72 6e 00 8a 01 5c 19 39 a4 43 8a 01
5c 3d 46 28 43 25 22;
secretKeysMap:
2017-05-17 18:44:53,392 DEBUG DelegationTokenAuthenticatedURL - credentials is
not empty. Fetching tokens now
2017-05-17 18:44:53,392 DEBUG DelegationTokenAuthenticatedURL -
serviceAddr=weichiu-foo-3.example.com/172.31.123.166:16000
text=172.31.123.166:16000
2017-05-17 18:44:53,392 DEBUG DelegationTokenAuthenticatedURL - dToken=null
2017-05-17 18:44:53,392 DEBUG DelegationTokenAuthenticator - do I have
delegation token? false
2017-05-17 18:44:53,427 DEBUG KerberosAuthenticator - Performing our own SPNEGO
sequence.
2017-05-17 18:44:53,428 DEBUG KerberosAuthenticator - No subject in context,
logging in
2017-05-17 18:44:53,428 DEBUG KerberosAuthenticator - Using subject: Subject:
Principal: UnixPrincipal: foo
Principal: UnixNumericUserPrincipal: 2004
Principal: UnixNumericGroupPrincipal [Primary Group]: 2004
2017-05-17 18:44:53,431 WARN Token - Cannot find class for token kind kms-dt
2017-05-17 18:44:53,431 DEBUG UserGroupInformation - PrivilegedActionException
as:foo (auth:KERBEROS) subject=Subject:
Principal: UnixPrincipal: foo
Principal: UnixNumericUserPrincipal: 2004
Principal: UnixNumericGroupPrincipal [Primary Group]: 2004
Principal: foo
Private Credential: tokenMap: key=172.31.117.206:8032 value=Kind:
RM_DELEGATION_TOKEN, Service: 172.31.117.206:8032, Ident: 00 18 61 74 74 69 76
69 6f 40 47 43 45 2e 43 4c 4f 55 44 45 52 41 2e 43 4f 4d 04 79 61 72 6e 00 8a
01 5c 19 39 a4 55 8a 01 5c 3d 46 28 55 1a 02;
key=ha-hdfs:ns1 value=Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:ns1, Ident:
(HDFS_DELEGATION_TOKEN token 110 for foo);
key=172.31.123.173:16000 value=Kind: kms-dt, Service: 172.31.123.173:16000,
Ident: 00 07 61 74 74 69 76 69 6f 04 79 61 72 6e 00 8a 01 5c 19 39 a4 43 8a 01
5c 3d 46 28 43 25 22;
secretKeysMap:
cause:org.apache.hadoop.security.authentication.client.AuthenticationException:
GSSException: No valid credentials provided (Mechanism level: Failed to find
any Kerberos tgt)
2017-05-17 18:44:53,431 WARN LoadBalancingKMSClientProvider - KMS provider at
[https://weichiu-foo-3.example.com:16000/kms/v1/] threw an IOException
[org.apache.hadoop.security.authentication.client.AuthenticationException:
GSSException: No valid credentials provided (Mechanism level: Failed to find
any Kerberos tgt)]!!
2017-05-17 18:44:53,433 WARN LoadBalancingKMSClientProvider -
stacktrace=java.io.IOException:
org.apache.hadoop.security.authentication.client.AuthenticationException:
GSSException: No valid credentials provided (Mechanism level: Failed to find
any Kerberos tgt)
at
org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:500)
at
org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:787)
at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$3.call(LoadBalancingKMSClientProvider.java:192)
at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$3.call(LoadBalancingKMSClientProvider.java:188)
at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:97)
at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.decryptEncryptedKey(LoadBalancingKMSClientProvider.java:188)
at
org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:388)
at
org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:1381)
at
org.apache.hadoop.hdfs.DFSClient.createWrappedInputStream(DFSClient.java:1451)
at
org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:305)
at
org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:299)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:312)foo
> LoadBalancingKMSClientProvider#addDelegationTokens should add delegation
> tokens from all KMS instances
> ------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-14441
> URL: https://issues.apache.org/jira/browse/HADOOP-14441
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
> Affects Versions: 2.7.0
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
> Reporter: Wei-Chiu Chuang
> Assignee: Wei-Chiu Chuang
> Attachments: HADOOP-14441.001.patch, HADOOP-14441.002.patch,
> HADOOP-14441.003.patch
>
>
> LoadBalancingKMSClientProvider only gets delegation token from one KMS
> instance, in a round-robin fashion. This is arguably a bug, as JavaDoc for
> {{KeyProviderDelegationTokenExtension#addDelegationTokens}} states:
> {quote}
> /**
> * The implementer of this class will take a renewer and add all
> * delegation tokens associated with the renewer to the
> * <code>Credentials</code> object if it is not already present,
> ...
> **/
> {quote}
> This bug doesn't pop up very often, because HDFS clients such as MapReduce
> unintentionally calls {{FileSystem#addDelegationTokens}} multiple times.
> We have a custom client that accesses HDFS/KMS-HA using delegation token, and
> we were puzzled why it always throws "Failed to find any Kerberos tgt"
> exceptions talking to one KMS but not the other. Turns out that client
> couldn't talk to the KMS because {{FileSystem#addDelegationTokens}} only gets
> one KMS delegation token at a time.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
