[
https://issues.apache.org/jira/browse/HADOOP-13173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuren Wu updated HADOOP-13173:
------------------------------
Description:
Hadoop cluster: multihomed network with KMS and Kerberos.
Sympton: mapreduce mapper will fail with kerberos error. (stack trace later)
Let's assume hadoop cluster has two networks, PRIVATE_NET and PUBLIC_NET.
When mapreduce job submitted to the cluster from an outside ndoe (node only has
visibility to PUBLIC_NET, it will acquire KMS tokens based on PUBLIC_NET ip
address. Such token in Credential's token list using IP will not be correct
resolved in the cluster. Therefore, we are using
hadoop.security.token.service.use_ip=false in the client side core-site.xml
file.
However, once job is accepted by yarn, the mapper container still throws
kerberos error. A detailed trace shows that when DFSClient is calling
decryptEncryptedDataencryptionKey function, it will reach to
org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection
method. This method will call SecurityUtil.buildtokenService to obtain the
service name and use it to lookup the token in Credentials.getToken(service)
call. This call returned the service in the IP:PORT format. Therefore, no token
can be retrieved from credential's token hashmap.
After read SecurityUtil class, I think this class does not load configuration
from configuration file at all. It seems to me that this class create a default
configuration object and the useIpForTokenService variable is always set false.
I do not have enough experience to provide a fix. In my mind, somewhere we need
to provide the correct value of useIpForTokenService based on job setup.
Stack trace provided here. this is from wordcount job from example.
2016-05-17 22:11:49,428 WARN [main] org.apache.hadoop.mapred.YarnChild:
Exception running child : java.io.IOException:
org.apache.hadoop.security.authentication.client.AuthenticationException:
GSSException: No valid credentials provided (Mechanism level: Failed to find
any Kerberos tgt)
at
org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:503)
at
org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:766)
at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$3.call(LoadBalancingKMSClientProvider.java:185)
at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$3.call(LoadBalancingKMSClientProvider.java:181)
at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:94)
at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.decryptEncryptedKey(LoadBalancingKMSClientProvider.java:181)
at
org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:388)
at
org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:1419)
at
org.apache.hadoop.hdfs.DFSClient.createWrappedInputStream(DFSClient.java:1489)
at
org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:310)
at
org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:304)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:304)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:775)
at
org.apache.hadoop.mapreduce.lib.input.LineRecordReader.initialize(LineRecordReader.java:85)
at
org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:548)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:786)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by:
org.apache.hadoop.security.authentication.client.AuthenticationException:
GSSException: No valid credentials provided (Mechanism level: Failed to find
any Kerberos tgt)
at
org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:318)
at
org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:203)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:127)
at
org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:216)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.openConnection(DelegationTokenAuthenticatedURL.java:373)
at
org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:497)
at
org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:493)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at
org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:493)
... 22 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed
to find any Kerberos tgt)
at
sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
at
sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
at
sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
at
sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)
at
sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
at
sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
at
org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:297)
at
org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:275)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:275)
... 32 more
was:
Hadoop cluster: multihomed network with KMS and Kerberos.
Sympton: mapreduce mapper will fail with kerberos error. (stack trace later)
Let's assume hadoop cluster has two networks, PRIVATE_NET and PUBLIC_NET.
When mapreduce job submitted to the cluster from an outside ndoe (node only has
visibility to PUBLIC_NET, it will acquire KMS tokens based on PUBLIC_NET ip
address. Such token in Credential's token list using IP will not be correct
resolved in the cluster. Therefore, we are using
hadoop.security.token.service.use_ip=false in the client side core-site.xml
file.
However, once job is accepted by yarn, the mapper container still throws
kerberos error. A detailed trace shows that when DFSClient is calling
decryptEncryptedDataencryptionKey function, it will reach to
org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection
method. This method will call SecurityUtil.buildtokenService to obtain the
service name and use it to lookup the token in Credentials.getToken(service)
call. This call returned the service in the IP:PORT format. Therefore, no token
can be retrieved from credential's token hashmap.
After read SecurityUtil class, I think this class does not load configuration
from configuration file at all. It seems to me that this class create a default
configuration object and the useIpForTokenService variable is always set false.
I do not have enough experience to provide a fix. In my mind, somewhere we need
to provide the correct value of useIpForTokenService based on job setup.
> SecurityUtil.buildtokenService broke token authentication in multi-home
> network env
> -----------------------------------------------------------------------------------
>
> Key: HADOOP-13173
> URL: https://issues.apache.org/jira/browse/HADOOP-13173
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
> Affects Versions: 2.6.0
> Environment: multi-homed network cluster on SLES 11
> Reporter: Yuren Wu
>
> Hadoop cluster: multihomed network with KMS and Kerberos.
> Sympton: mapreduce mapper will fail with kerberos error. (stack trace later)
> Let's assume hadoop cluster has two networks, PRIVATE_NET and PUBLIC_NET.
> When mapreduce job submitted to the cluster from an outside ndoe (node only
> has visibility to PUBLIC_NET, it will acquire KMS tokens based on PUBLIC_NET
> ip address. Such token in Credential's token list using IP will not be
> correct resolved in the cluster. Therefore, we are using
> hadoop.security.token.service.use_ip=false in the client side core-site.xml
> file.
> However, once job is accepted by yarn, the mapper container still throws
> kerberos error. A detailed trace shows that when DFSClient is calling
> decryptEncryptedDataencryptionKey function, it will reach to
> org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection
> method. This method will call SecurityUtil.buildtokenService to obtain the
> service name and use it to lookup the token in Credentials.getToken(service)
> call. This call returned the service in the IP:PORT format. Therefore, no
> token can be retrieved from credential's token hashmap.
> After read SecurityUtil class, I think this class does not load configuration
> from configuration file at all. It seems to me that this class create a
> default configuration object and the useIpForTokenService variable is always
> set false.
> I do not have enough experience to provide a fix. In my mind, somewhere we
> need to provide the correct value of useIpForTokenService based on job setup.
> Stack trace provided here. this is from wordcount job from example.
> 2016-05-17 22:11:49,428 WARN [main] org.apache.hadoop.mapred.YarnChild:
> Exception running child : java.io.IOException:
> org.apache.hadoop.security.authentication.client.AuthenticationException:
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:503)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:766)
> at
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$3.call(LoadBalancingKMSClientProvider.java:185)
> at
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$3.call(LoadBalancingKMSClientProvider.java:181)
> at
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:94)
> at
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.decryptEncryptedKey(LoadBalancingKMSClientProvider.java:181)
> at
> org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:388)
> at
> org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:1419)
> at
> org.apache.hadoop.hdfs.DFSClient.createWrappedInputStream(DFSClient.java:1489)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:310)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:304)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:304)
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:775)
> at
> org.apache.hadoop.mapreduce.lib.input.LineRecordReader.initialize(LineRecordReader.java:85)
> at
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:548)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:786)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by:
> org.apache.hadoop.security.authentication.client.AuthenticationException:
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)
> at
> org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:318)
> at
> org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:203)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:127)
> at
> org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:216)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.openConnection(DelegationTokenAuthenticatedURL.java:373)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:497)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:493)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:493)
> ... 22 more
> Caused by: GSSException: No valid credentials provided (Mechanism level:
> Failed to find any Kerberos tgt)
> at
> sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
> at
> sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
> at
> sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
> at
> sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)
> at
> sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
> at
> sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
> at
> org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:297)
> at
> org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:275)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:275)
> ... 32 more
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
