[jira] [Comment Edited] (HDFS-13682) Cannot create encryption zone after KMS auth token expires
[
https://issues.apache.org/jira/browse/HDFS-13682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513312#comment-16513312
]
Xiao Chen edited comment on HDFS-13682 at 6/15/18 4:08 PM:
---
Took an easier route and debugged branch-2. It turns out HADOOP-9747 does have
some effects here - specifically at [this
method|https://github.com/apache/hadoop/commit/59cf7588779145ad5850ad63426743dfe03d8347#diff-e6a2371b73365b7ba7ff9a266b9aa138L724].
When this meets the KMSCP's morph-based-on-ugi logic, the ugi being used as
actual changed from loginUgi to currentUgi. (Also has a weird HTTP 400 somehow,
which is fixed if contentType is not empty).
Following this, I confirmed if we change {{KMSCP#getActualUgi}}'s check from
{{actualUgi.hasKerberosCredentials()}} to {{!actualUgi.isFromKeytab() &&
!actualUgi.isFromTicket()}} (and making {{UGI#isFromTicket}} public of course),
the test passes. This appears to be a more 'compatible' change. Patch 1 tries
to do this.
IMO we should still consider explicitly doing the KMS calls in the NN using the
NN login ugi, this applies to both the {{getMetadata}} call during createEZ and
the {{generateEncryptedKey}} call from startFile. Reason being these calls are
internal to the NN, and the hdfs rpc caller isn't expected to really interact
with the KMS in these cases. Can do this in a separate Jira if it sounds good
to the audience.
was (Author: xiaochen):
Took an easier route and debugged branch-2. It turns out HADOOP-9747 does have
some effects here - specifically at [this
method|https://github.com/apache/hadoop/commit/59cf7588779145ad5850ad63426743dfe03d8347#diff-e6a2371b73365b7ba7ff9a266b9aa138L724].
When this meets the KMSCP's morph-based-on-ugi logic, the ugi being used as
actual changed from loginUgi to currentUgi. (Also has a weird HTTP 400 somehow,
which is fixed if contentType is set).
Following this, I confirmed if we change {{KMSCP#getActualUgi}}'s check from
{{actualUgi.hasKerberosCredentials()}} to {{!actualUgi.isFromKeytab() &&
!actualUgi.isFromTicket()}} (and making {{UGI#isFromTicket}} public of course),
the test passes. This appears to be a more 'compatible' change. Patch 1 tries
to do this.
IMO we should still consider explicitly doing the KMS call using the NN login
ugi, this applies to both the {{getMetadata}} call during createEZ and the
{{generateEncryptedKey}} call from startFile. Reason being these calls are
internal to the NN, and the hdfs rpc caller isn't expected to really interact
with the KMS in this case. Can do this in a separate Jira if it sounds good to
the audience.
> Cannot create encryption zone after KMS auth token expires
> --
>
> Key: HDFS-13682
> URL: https://issues.apache.org/jira/browse/HDFS-13682
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption, namenode
>Affects Versions: 3.0.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Critical
> Attachments: HDFS-13682.01.patch,
> HDFS-13682.dirty.repro.branch-2.patch, HDFS-13682.dirty.repro.patch
>
>
> Our internal testing reported this behavior recently.
> {noformat}
> [root@nightly6x-1 ~]# sudo -u hdfs /usr/bin/kinit -kt
> /cdep/keytabs/hdfs.keytab hdfs -l 30d -r 30d
> [root@nightly6x-1 ~]# sudo -u hdfs klist
> Ticket cache: FILE:/tmp/krb5cc_994
> Default principal: h...@gce.cloudera.com
> Valid starting Expires Service principal
> 06/12/2018 03:24:09 07/12/2018 03:24:09
> krbtgt/gce.cloudera@gce.cloudera.com
> [root@nightly6x-1 ~]# sudo -u hdfs hdfs crypto -createZone -keyName key77
> -path /user/systest/ez
> RemoteException:
> org.apache.hadoop.security.authentication.client.AuthenticationException:
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)
> {noformat}
> Upon further investigation, it's due to the KMS client (cached in HDFS NN)
> cannot authenticate with the server after the authentication token (which is
> cached by KMSCP) expires, even if the HDFS client RPC has valid kerberos
> credentials.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13682) Cannot create encryption zone after KMS auth token expires
[
https://issues.apache.org/jira/browse/HDFS-13682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513312#comment-16513312
]
Xiao Chen edited comment on HDFS-13682 at 6/15/18 4:58 AM:
---
Took an easier route and debugged branch-2. It turns out HADOOP-9747 does have
some effects here - specifically at [this
method|https://github.com/apache/hadoop/commit/59cf7588779145ad5850ad63426743dfe03d8347#diff-e6a2371b73365b7ba7ff9a266b9aa138L724].
When this meets the KMSCP's morph-based-on-ugi logic, the ugi being used as
actual changed from loginUgi to currentUgi. (Also has a weird HTTP 400 somehow,
which is fixed if contentType is set).
Following this, I confirmed if we change {{KMSCP#getActualUgi}}'s check from
{{actualUgi.hasKerberosCredentials()}} to {{!actualUgi.isFromKeytab() &&
!actualUgi.isFromTicket()}} (and making {{UGI#isFromTicket}} public of course),
the test passes. This appears to be a more 'compatible' change. Patch 1 tries
to do this.
IMO we should still consider explicitly doing the KMS call using the NN login
ugi, this applies to both the {{getMetadata}} call during createEZ and the
{{generateEncryptedKey}} call from startFile. Reason being these calls are
internal to the NN, and the hdfs rpc caller isn't expected to really interact
with the KMS in this case. Can do this in a separate Jira if it sounds good to
the audience.
was (Author: xiaochen):
Took an easier route and debugged branch-2. It turns out HADOOP-9747 does have
some effects here - specifically at [this
method|https://github.com/apache/hadoop/commit/59cf7588779145ad5850ad63426743dfe03d8347#diff-e6a2371b73365b7ba7ff9a266b9aa138L724].
When this meets the KMSCP's morph-based-on-ugi logic, the ugi being used as
actual changed from loginUgi to currentUgi. (Also has a weird HTTP 400 somehow,
which is fixed if contentType is set).
Following this, I confirmed if we change {{KMSCP#getActualUgi}}'s check from
{{actualUgi.hasKerberosCredentials()}} to {{!actualUgi.isFromKeytab() &&
!actualUgi.isFromTicket()}} (and making {{UGI#isFromTicket}} public of course),
the test passes. This appears to be a more 'compatible' change.
IMO we should still consider explicitly doing the KMS call using the NN login
ugi, this applies to both the {{getMetadata}} call during createEZ and the
{{generateEncryptedKey}} call from startFile. Reason being these calls are
internal to the NN, and the hdfs rpc caller isn't expected to really interact
with the KMS in this case. Can do this in a separate Jira if it sounds good to
the audience.
> Cannot create encryption zone after KMS auth token expires
> --
>
> Key: HDFS-13682
> URL: https://issues.apache.org/jira/browse/HDFS-13682
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption, namenode
>Affects Versions: 3.0.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Critical
> Attachments: HDFS-13682.01.patch,
> HDFS-13682.dirty.repro.branch-2.patch, HDFS-13682.dirty.repro.patch
>
>
> Our internal testing reported this behavior recently.
> {noformat}
> [root@nightly6x-1 ~]# sudo -u hdfs /usr/bin/kinit -kt
> /cdep/keytabs/hdfs.keytab hdfs -l 30d -r 30d
> [root@nightly6x-1 ~]# sudo -u hdfs klist
> Ticket cache: FILE:/tmp/krb5cc_994
> Default principal: h...@gce.cloudera.com
> Valid starting Expires Service principal
> 06/12/2018 03:24:09 07/12/2018 03:24:09
> krbtgt/gce.cloudera@gce.cloudera.com
> [root@nightly6x-1 ~]# sudo -u hdfs hdfs crypto -createZone -keyName key77
> -path /user/systest/ez
> RemoteException:
> org.apache.hadoop.security.authentication.client.AuthenticationException:
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)
> {noformat}
> Upon further investigation, it's due to the KMS client (cached in HDFS NN)
> cannot authenticate with the server after the authentication token (which is
> cached by KMSCP) expires, even if the HDFS client RPC has valid kerberos
> credentials.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13682) Cannot create encryption zone after KMS auth token expires
[
https://issues.apache.org/jira/browse/HDFS-13682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513312#comment-16513312
]
Xiao Chen edited comment on HDFS-13682 at 6/15/18 4:08 AM:
---
Took an easier route and debugged branch-2. It turns out HADOOP-9747 does have
some effects here - specifically at [this
method|https://github.com/apache/hadoop/commit/59cf7588779145ad5850ad63426743dfe03d8347#diff-e6a2371b73365b7ba7ff9a266b9aa138L724].
When this meets the KMSCP's morph-based-on-ugi logic, the ugi being used as
actual changed from loginUgi to currentUgi. (Also has a weird HTTP 400 somehow,
which is fixed if contentType is set).
Following this, I confirmed if we change {{KMSCP#getActualUgi}}'s check from
{{actualUgi.hasKerberosCredentials()}} to {{!actualUgi.isFromKeytab() &&
!actualUgi.isFromTicket()}} (and making {{UGI#isFromTicket}} public of course),
the test passes. This appears to be a more 'compatible' change.
IMO we should still consider explicitly doing the KMS call using the NN login
ugi, this applies to both the {{getMetadata}} call during createEZ and the
{{generateEncryptedKey}} call from startFile. Reason being these calls are
internal to the NN, and the hdfs rpc caller isn't expected to really interact
with the KMS in this case. Can do this in a separate Jira if it sounds good to
the audience.
was (Author: xiaochen):
Took an easier route and debugged branch-2. It turns out HADOOP-9747 does have
some effects here - specifically at [this
method|https://github.com/apache/hadoop/commit/59cf7588779145ad5850ad63426743dfe03d8347#diff-e6a2371b73365b7ba7ff9a266b9aa138L724].
When this meets the KMSCP's morph-based-on-ugi logic, the ugi being used as
actual changed from loginUgi to currentUgi. (Also has a weird HTTP 400 somehow,
which is fixed if contentType is set).
Following this, I confirmed if we change {{KMSCP#getActualUgi}}'s check from
{{actualUgi.hasKerberosCredentials()}} to {{!actualUgi.isFromKeytab() &&
!actualUgi.isFromTicket()}} (and making {{UGI#isFromTicket}} public of course),
the test passes. This appears to be a more 'compatible' change.
IMO we should still consider explicitly doing the KMS call using the NN login
ugi, this applies to both the {{getMetadata}} call during createEZ and the
\{{generateEncryptedKey}} call from startFile. Reason being these calls are
internal to the NN, and the hdfs rpc caller isn't expected to really interact
with the KMS in this case.
> Cannot create encryption zone after KMS auth token expires
> --
>
> Key: HDFS-13682
> URL: https://issues.apache.org/jira/browse/HDFS-13682
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption, namenode
>Affects Versions: 3.0.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Critical
> Attachments: HDFS-13682.dirty.repro.branch-2.patch,
> HDFS-13682.dirty.repro.patch
>
>
> Our internal testing reported this behavior recently.
> {noformat}
> [root@nightly6x-1 ~]# sudo -u hdfs /usr/bin/kinit -kt
> /cdep/keytabs/hdfs.keytab hdfs -l 30d -r 30d
> [root@nightly6x-1 ~]# sudo -u hdfs klist
> Ticket cache: FILE:/tmp/krb5cc_994
> Default principal: h...@gce.cloudera.com
> Valid starting Expires Service principal
> 06/12/2018 03:24:09 07/12/2018 03:24:09
> krbtgt/gce.cloudera@gce.cloudera.com
> [root@nightly6x-1 ~]# sudo -u hdfs hdfs crypto -createZone -keyName key77
> -path /user/systest/ez
> RemoteException:
> org.apache.hadoop.security.authentication.client.AuthenticationException:
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)
> {noformat}
> Upon further investigation, it's due to the KMS client (cached in HDFS NN)
> cannot authenticate with the server after the authentication token (which is
> cached by KMSCP) expires, even if the HDFS client RPC has valid kerberos
> credentials.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
