[
https://issues.apache.org/jira/browse/HADOOP-15414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16454363#comment-16454363
]
Daryn Sharp commented on HADOOP-15414:
--------------------------------------
That patch is not going to work correctly. It's likely to cause unintended
recursive issues and just make token collection even messier.
The root cause of this jira is {{DistributedFileSystem}} bent the
(undocumented) rules for acquiring the kms token by overriding
{{FileSystem#addDelegationTokens}}.
The way it's supposed to work is a filesystem implements {{getDelegationToken}}
to acquire its specific token. If the filesystem needs additional ancillary
tokens, {{getChildFileSystems}} should return a list of those filesystems which
in turn may implement {{getDelegationToken}} and maybe {{getChildFileSystems}}.
This pattern was added specifically to support viewfs.
So why is EZ broken through viewfs? {{FileSystem#addDelegationTokens}} via
{{collectDelegationTokens}} will recursively invoke {{getDelegationToken}} on
each filesystem and its children if it doesn't already have a token for the
filesystem – _not_ {{addDelegationTokens}} which is what
{{DistributedFileSystem}} has overridden to add the kms token. This in turn
caused the kms client provider to copy the chunk of code that determined if a
token was required and how to insert in the credentials.
Back when I added these methods to support viewfs, here's what I really wanted
to do:
* make {{FileSystem#addDelegationTokens}} a final method (would have prevented
this bug, but too late now)
* add a {{TokenIssuer}} interface for get/cancel/renew/canonical service methods
* {{getChildFileSystems}} should have been {{getTokenIssuers}} to allow
filesystems to acquire non-filesystem tokens. ie. the kms client!
That design would allow the kms client to implement {{TokenIssuer}}, and
{{DistributedFileSystem#getTokenIssuers}} to return the kms client, and for kms
client to delete a big chunk of code. I think we can still do this. The
default {{getTokenIssuers}} would return {{getChildFileSystems}}.
I can mock up a POC if this doesn't make sense.
> Job submit not work well on HDFS Federation with Transparent Encryption
> feature
> -------------------------------------------------------------------------------
>
> Key: HADOOP-15414
> URL: https://issues.apache.org/jira/browse/HADOOP-15414
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Reporter: He Xiaoqiao
> Priority: Major
> Attachments: HADOOP-15414-trunk.001.patch
>
>
> When submit sample MapReduce job WordCount which read/write path under
> encryption zone on HDFS Federation in security mode to YARN, task throws
> exception as below:
> {code:java}
> 18/04/26 16:07:26 INFO mapreduce.Job: Task Id : attempt_JOBID_m_TASKID_0,
> Status : FAILED
> Error: java.io.IOException:
> org.apache.hadoop.security.authentication.client.AuthenticationException:
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:489)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:776)
> at
> org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:388)
> at
> org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:1468)
> at
> org.apache.hadoop.hdfs.DFSClient.createWrappedInputStream(DFSClient.java:1538)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:306)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:300)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:300)
> at org.apache.hadoop.fs.FilterFileSystem.open(FilterFileSystem.java:161)
> at
> org.apache.hadoop.fs.viewfs.ChRootedFileSystem.open(ChRootedFileSystem.java:258)
> at
> org.apache.hadoop.fs.viewfs.ViewFileSystem.open(ViewFileSystem.java:424)
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:793)
> at
> org.apache.hadoop.mapreduce.lib.input.LineRecordReader.initialize(LineRecordReader.java:85)
> at
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:552)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:823)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1690)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
> Caused by:
> org.apache.hadoop.security.authentication.client.AuthenticationException:
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)
> at
> org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:332)
> at
> org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:205)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:128)
> at
> org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:215)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.openConnection(DelegationTokenAuthenticatedURL.java:322)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:483)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:478)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1690)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:478)
> ... 21 more
> Caused by: GSSException: No valid credentials provided (Mechanism level:
> Failed to find any Kerberos tgt)
> at
> sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
> at
> sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
> at
> sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
> at
> sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
> at
> sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
> at
> sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
> at
> org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:311)
> at
> org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:287)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:287)
> ... 31 more
> {code}
> The main reason is before submit job in security mode, we need to collect
> delegation tokens include delegation token for NameNode and KMS firstly. IF
> on HDFS Federation, all delegation tokens for NameNode can collection
> correctly BUT delegation token for KMS not collect reference
> {{FileSystem#addDelegationTokens}} -> {{FileSystem#collectDelegationTokens}},
> so when launch task it fails because KMS token not pass to through
> ResourceManager as exception shows {{GSSException: No valid credentials
> provided}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
