[ https://issues.apache.org/jira/browse/HDFS-13682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16513312#comment-16513312 ]
Xiao Chen edited comment on HDFS-13682 at 6/15/18 4:08 PM: ----------------------------------------------------------- Took an easier route and debugged branch-2. It turns out HADOOP-9747 does have some effects here - specifically at [this method|https://github.com/apache/hadoop/commit/59cf7588779145ad5850ad63426743dfe03d8347#diff-e6a2371b73365b7ba7ff9a266b9aa138L724]. When this meets the KMSCP's morph-based-on-ugi logic, the ugi being used as actual changed from loginUgi to currentUgi. (Also has a weird HTTP 400 somehow, which is fixed if contentType is not empty). Following this, I confirmed if we change {{KMSCP#getActualUgi}}'s check from {{actualUgi.hasKerberosCredentials()}} to {{!actualUgi.isFromKeytab() && !actualUgi.isFromTicket()}} (and making {{UGI#isFromTicket}} public of course), the test passes. This appears to be a more 'compatible' change. Patch 1 tries to do this. IMO we should still consider explicitly doing the KMS calls in the NN using the NN login ugi, this applies to both the {{getMetadata}} call during createEZ and the {{generateEncryptedKey}} call from startFile. Reason being these calls are internal to the NN, and the hdfs rpc caller isn't expected to really interact with the KMS in these cases. Can do this in a separate Jira if it sounds good to the audience. was (Author: xiaochen): Took an easier route and debugged branch-2. It turns out HADOOP-9747 does have some effects here - specifically at [this method|https://github.com/apache/hadoop/commit/59cf7588779145ad5850ad63426743dfe03d8347#diff-e6a2371b73365b7ba7ff9a266b9aa138L724]. When this meets the KMSCP's morph-based-on-ugi logic, the ugi being used as actual changed from loginUgi to currentUgi. (Also has a weird HTTP 400 somehow, which is fixed if contentType is set). Following this, I confirmed if we change {{KMSCP#getActualUgi}}'s check from {{actualUgi.hasKerberosCredentials()}} to {{!actualUgi.isFromKeytab() && !actualUgi.isFromTicket()}} (and making {{UGI#isFromTicket}} public of course), the test passes. This appears to be a more 'compatible' change. Patch 1 tries to do this. IMO we should still consider explicitly doing the KMS call using the NN login ugi, this applies to both the {{getMetadata}} call during createEZ and the {{generateEncryptedKey}} call from startFile. Reason being these calls are internal to the NN, and the hdfs rpc caller isn't expected to really interact with the KMS in this case. Can do this in a separate Jira if it sounds good to the audience. > Cannot create encryption zone after KMS auth token expires > ---------------------------------------------------------- > > Key: HDFS-13682 > URL: https://issues.apache.org/jira/browse/HDFS-13682 > Project: Hadoop HDFS > Issue Type: Bug > Components: encryption, namenode > Affects Versions: 3.0.0 > Reporter: Xiao Chen > Assignee: Xiao Chen > Priority: Critical > Attachments: HDFS-13682.01.patch, > HDFS-13682.dirty.repro.branch-2.patch, HDFS-13682.dirty.repro.patch > > > Our internal testing reported this behavior recently. > {noformat} > [root@nightly6x-1 ~]# sudo -u hdfs /usr/bin/kinit -kt > /cdep/keytabs/hdfs.keytab hdfs -l 30d -r 30d > [root@nightly6x-1 ~]# sudo -u hdfs klist > Ticket cache: FILE:/tmp/krb5cc_994 > Default principal: h...@gce.cloudera.com > Valid starting Expires Service principal > 06/12/2018 03:24:09 07/12/2018 03:24:09 > krbtgt/gce.cloudera....@gce.cloudera.com > [root@nightly6x-1 ~]# sudo -u hdfs hdfs crypto -createZone -keyName key77 > -path /user/systest/ez > RemoteException: > org.apache.hadoop.security.authentication.client.AuthenticationException: > GSSException: No valid credentials provided (Mechanism level: Failed to find > any Kerberos tgt) > {noformat} > Upon further investigation, it's due to the KMS client (cached in HDFS NN) > cannot authenticate with the server after the authentication token (which is > cached by KMSCP) expires, even if the HDFS client RPC has valid kerberos > credentials. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org