[jira] [Commented] (HADOOP-14441) LoadBalancingKMSClientProvider#addDelegationTokens should add delegation tokens from all KMS instances

[Wei-Chiu Chuang (Jira)]

[ 
https://issues.apache.org/jira/browse/HADOOP-14441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016433#comment-17016433
 ] 

Wei-Chiu Chuang commented on HADOOP-14441:
--

HADOOP-14445 is an extensive change and getting it into Hadoop 2.x is going to 
be require some good amount of cycles to get done. I am hesitate to backport 
HADOOP-14445 to 2.x.

For any one on 2.x that still want to resolve this issue, HADOOP-1441 is a 
simpler approach to address the issue -- There was a bug in the 004 patch so 
attached a 005 patch to update it. 

This "simpler" approach has 2 downsides:
(1) it acquires one delegation token from each KMS instance. So the number of 
delegation tokens increases w.r.t to the number of KMS. In a busy/big cluster 
it can grow so much that Zookeeper (the delegation token store) is overwhelmed.
(2) if an application acquires delegation token when KMS1 is down, it will only 
acquire the dt from KMS2. Later if KMS1 comes back but KMS2 goes down, this 
application will fail to access KMS. It is a likely scenario during cluster 
rolling restart.

> LoadBalancingKMSClientProvider#addDelegationTokens should add delegation 
> tokens from all KMS instances
> --
>
> Key: HADOOP-14441
> URL: https://issues.apache.org/jira/browse/HADOOP-14441
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 2.7.0
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Attachments: HADOOP-14441.001.patch, HADOOP-14441.002.patch, 
> HADOOP-14441.003.patch, HADOOP-14441.004.patch, 
> HADOOP-14441.branch-2.005.patch
>
>
> LoadBalancingKMSClientProvider only gets delegation token from one KMS 
> instance, in a round-robin fashion. This is arguably a bug, as JavaDoc for 
> {{KeyProviderDelegationTokenExtension#addDelegationTokens}} states:
> {quote}
> /**
>  * The implementer of this class will take a renewer and add all
>  * delegation tokens associated with the renewer to the 
>  * Credentials object if it is not already present, 
> ...
> **/
> {quote}
> This bug doesn't pop up very often, because HDFS clients such as MapReduce 
> unintentionally calls {{FileSystem#addDelegationTokens}} multiple times.
> We have a custom client that accesses HDFS/KMS-HA using delegation token, and 
> we were puzzled why it always throws "Failed to find any Kerberos tgt" 
> exceptions talking to one KMS but not the other. Turns out that client 
> couldn't talk to the KMS because {{FileSystem#addDelegationTokens}} only gets 
> one KMS delegation token at a time.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-14441) LoadBalancingKMSClientProvider#addDelegationTokens should add delegation tokens from all KMS instances

[Wei-Chiu Chuang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-14441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904559#comment-16904559
 ] 

Wei-Chiu Chuang commented on HADOOP-14441:
--

Try to apply  [^HADOOP-14441.004.patch]. This was patched against trunk, so 
it's unlikely to apply cleanly to 2.7.x.

Another way is to update KMS server to 2.8.x or above. The change in 
HADOOP-14445 (which is the proper fix for this bug) is supposed to be backward 
compatible.

> LoadBalancingKMSClientProvider#addDelegationTokens should add delegation 
> tokens from all KMS instances
> --
>
> Key: HADOOP-14441
> URL: https://issues.apache.org/jira/browse/HADOOP-14441
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 2.7.0
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Attachments: HADOOP-14441.001.patch, HADOOP-14441.002.patch, 
> HADOOP-14441.003.patch, HADOOP-14441.004.patch
>
>
> LoadBalancingKMSClientProvider only gets delegation token from one KMS 
> instance, in a round-robin fashion. This is arguably a bug, as JavaDoc for 
> {{KeyProviderDelegationTokenExtension#addDelegationTokens}} states:
> {quote}
> /**
>  * The implementer of this class will take a renewer and add all
>  * delegation tokens associated with the renewer to the 
>  * Credentials object if it is not already present, 
> ...
> **/
> {quote}
> This bug doesn't pop up very often, because HDFS clients such as MapReduce 
> unintentionally calls {{FileSystem#addDelegationTokens}} multiple times.
> We have a custom client that accesses HDFS/KMS-HA using delegation token, and 
> we were puzzled why it always throws "Failed to find any Kerberos tgt" 
> exceptions talking to one KMS but not the other. Turns out that client 
> couldn't talk to the KMS because {{FileSystem#addDelegationTokens}} only gets 
> one KMS delegation token at a time.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-14441) LoadBalancingKMSClientProvider#addDelegationTokens should add delegation tokens from all KMS instances

[Guido Aulisi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-14441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904442#comment-16904442
 ] 

Guido Aulisi commented on HADOOP-14441:
---

Thanks [~jojochuang] for your answer.

I usually hit this in Spark2 application from executors, access to encrypted 
files is really transparent from the application. So I don't know how to call 
addDelegationTokens from Spark2 applications.

> LoadBalancingKMSClientProvider#addDelegationTokens should add delegation 
> tokens from all KMS instances
> --
>
> Key: HADOOP-14441
> URL: https://issues.apache.org/jira/browse/HADOOP-14441
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 2.7.0
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Attachments: HADOOP-14441.001.patch, HADOOP-14441.002.patch, 
> HADOOP-14441.003.patch, HADOOP-14441.004.patch
>
>
> LoadBalancingKMSClientProvider only gets delegation token from one KMS 
> instance, in a round-robin fashion. This is arguably a bug, as JavaDoc for 
> {{KeyProviderDelegationTokenExtension#addDelegationTokens}} states:
> {quote}
> /**
>  * The implementer of this class will take a renewer and add all
>  * delegation tokens associated with the renewer to the 
>  * Credentials object if it is not already present, 
> ...
> **/
> {quote}
> This bug doesn't pop up very often, because HDFS clients such as MapReduce 
> unintentionally calls {{FileSystem#addDelegationTokens}} multiple times.
> We have a custom client that accesses HDFS/KMS-HA using delegation token, and 
> we were puzzled why it always throws "Failed to find any Kerberos tgt" 
> exceptions talking to one KMS but not the other. Turns out that client 
> couldn't talk to the KMS because {{FileSystem#addDelegationTokens}} only gets 
> one KMS delegation token at a time.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-14441) LoadBalancingKMSClientProvider#addDelegationTokens should add delegation tokens from all KMS instances

[Wei-Chiu Chuang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-14441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904264#comment-16904264
 ] 

Wei-Chiu Chuang commented on HADOOP-14441:
--

[~tartina] depending on your applications,
you could apply this patch to your Hadoop installation (2.7 is EOL, I think), 
or call FileSystem.addDelegationTokens() multiple times (depending on the 
number of KMS instances) to get delegation tokens from all KMS.

This is actually not a problem for MapReduce, because MR unintentionally calls 
FileSystem.addDelegationTokens() three times.

> LoadBalancingKMSClientProvider#addDelegationTokens should add delegation 
> tokens from all KMS instances
> --
>
> Key: HADOOP-14441
> URL: https://issues.apache.org/jira/browse/HADOOP-14441
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 2.7.0
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Attachments: HADOOP-14441.001.patch, HADOOP-14441.002.patch, 
> HADOOP-14441.003.patch, HADOOP-14441.004.patch
>
>
> LoadBalancingKMSClientProvider only gets delegation token from one KMS 
> instance, in a round-robin fashion. This is arguably a bug, as JavaDoc for 
> {{KeyProviderDelegationTokenExtension#addDelegationTokens}} states:
> {quote}
> /**
>  * The implementer of this class will take a renewer and add all
>  * delegation tokens associated with the renewer to the 
>  * Credentials object if it is not already present, 
> ...
> **/
> {quote}
> This bug doesn't pop up very often, because HDFS clients such as MapReduce 
> unintentionally calls {{FileSystem#addDelegationTokens}} multiple times.
> We have a custom client that accesses HDFS/KMS-HA using delegation token, and 
> we were puzzled why it always throws "Failed to find any Kerberos tgt" 
> exceptions talking to one KMS but not the other. Turns out that client 
> couldn't talk to the KMS because {{FileSystem#addDelegationTokens}} only gets 
> one KMS delegation token at a time.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-14441) LoadBalancingKMSClientProvider#addDelegationTokens should add delegation tokens from all KMS instances

[Guido Aulisi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-14441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903738#comment-16903738
 ] 

Guido Aulisi commented on HADOOP-14441:
---

Is there a workaround or patch for this issue for hadoop 2.7.3?

> LoadBalancingKMSClientProvider#addDelegationTokens should add delegation 
> tokens from all KMS instances
> --
>
> Key: HADOOP-14441
> URL: https://issues.apache.org/jira/browse/HADOOP-14441
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 2.7.0
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Attachments: HADOOP-14441.001.patch, HADOOP-14441.002.patch, 
> HADOOP-14441.003.patch, HADOOP-14441.004.patch
>
>
> LoadBalancingKMSClientProvider only gets delegation token from one KMS 
> instance, in a round-robin fashion. This is arguably a bug, as JavaDoc for 
> {{KeyProviderDelegationTokenExtension#addDelegationTokens}} states:
> {quote}
> /**
>  * The implementer of this class will take a renewer and add all
>  * delegation tokens associated with the renewer to the 
>  * Credentials object if it is not already present, 
> ...
> **/
> {quote}
> This bug doesn't pop up very often, because HDFS clients such as MapReduce 
> unintentionally calls {{FileSystem#addDelegationTokens}} multiple times.
> We have a custom client that accesses HDFS/KMS-HA using delegation token, and 
> we were puzzled why it always throws "Failed to find any Kerberos tgt" 
> exceptions talking to one KMS but not the other. Turns out that client 
> couldn't talk to the KMS because {{FileSystem#addDelegationTokens}} only gets 
> one KMS delegation token at a time.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-14441) LoadBalancingKMSClientProvider#addDelegationTokens should add delegation tokens from all KMS instances

[Rushabh S Shah (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-14441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16026825#comment-16026825
 ] 

Rushabh S Shah commented on HADOOP-14441:
-

bq. Would you mind posting your patch to HADOOP-14445 so that we can iterate?
[~yzhangal]: Posted branch-2.8 patch on HADOOP-14445.
Will appreciate your feedback.

> LoadBalancingKMSClientProvider#addDelegationTokens should add delegation 
> tokens from all KMS instances
> --
>
> Key: HADOOP-14441
> URL: https://issues.apache.org/jira/browse/HADOOP-14441
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 2.7.0
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Attachments: HADOOP-14441.001.patch, HADOOP-14441.002.patch, 
> HADOOP-14441.003.patch, HADOOP-14441.004.patch
>
>
> LoadBalancingKMSClientProvider only gets delegation token from one KMS 
> instance, in a round-robin fashion. This is arguably a bug, as JavaDoc for 
> {{KeyProviderDelegationTokenExtension#addDelegationTokens}} states:
> {quote}
> /**
>  * The implementer of this class will take a renewer and add all
>  * delegation tokens associated with the renewer to the 
>  * Credentials object if it is not already present, 
> ...
> **/
> {quote}
> This bug doesn't pop up very often, because HDFS clients such as MapReduce 
> unintentionally calls {{FileSystem#addDelegationTokens}} multiple times.
> We have a custom client that accesses HDFS/KMS-HA using delegation token, and 
> we were puzzled why it always throws "Failed to find any Kerberos tgt" 
> exceptions talking to one KMS but not the other. Turns out that client 
> couldn't talk to the KMS because {{FileSystem#addDelegationTokens}} only gets 
> one KMS delegation token at a time.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-14441) LoadBalancingKMSClientProvider#addDelegationTokens should add delegation tokens from all KMS instances

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-14441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025254#comment-16025254
 ] 

Hadoop QA commented on HADOOP-14441:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
7s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m 
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m 
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
45s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  1m 
18s{color} | {color:red} hadoop-common-project/hadoop-common in trunk has 19 
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
7s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m 
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m 
39s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 34s{color} | {color:orange} hadoop-common-project: The patch generated 4 new 
+ 100 unchanged - 3 fixed = 104 total (was 103) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  1m 
35s{color} | {color:red} hadoop-common-project/hadoop-common generated 1 new + 
19 unchanged - 0 fixed = 20 total (was 19) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
2s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  7m  1s{color} 
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 
47s{color} | {color:green} hadoop-kms in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
26s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 65m 11s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-common-project/hadoop-common |
|  |  instanceof will always return true for all non-null values in 
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.addDelegationTokens(String,
 Credentials), since all RuntimeException are instances of RuntimeException  At 
LoadBalancingKMSClientProvider.java:for all non-null values in 
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.addDelegationTokens(String,
 Credentials), since all RuntimeException are instances of RuntimeException  At 
LoadBalancingKMSClientProvider.java:[line 154] |
| Failed junit tests | hadoop.fs.sftp.TestSFTPFileSystem |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HADOOP-14441 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12869902/HADOOP-14441.004.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 

[jira] [Commented] (HADOOP-14441) LoadBalancingKMSClientProvider#addDelegationTokens should add delegation tokens from all KMS instances

[Yongjun Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-14441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025088#comment-16025088
 ] 

Yongjun Zhang commented on HADOOP-14441:


HI [~shahrs87],

Would you mind posting your patch to HADOOP-14445 so that we can iterate? 

Thanks a lot.


> LoadBalancingKMSClientProvider#addDelegationTokens should add delegation 
> tokens from all KMS instances
> --
>
> Key: HADOOP-14441
> URL: https://issues.apache.org/jira/browse/HADOOP-14441
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 2.7.0
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Attachments: HADOOP-14441.001.patch, HADOOP-14441.002.patch, 
> HADOOP-14441.003.patch, HADOOP-14441.004.patch
>
>
> LoadBalancingKMSClientProvider only gets delegation token from one KMS 
> instance, in a round-robin fashion. This is arguably a bug, as JavaDoc for 
> {{KeyProviderDelegationTokenExtension#addDelegationTokens}} states:
> {quote}
> /**
>  * The implementer of this class will take a renewer and add all
>  * delegation tokens associated with the renewer to the 
>  * Credentials object if it is not already present, 
> ...
> **/
> {quote}
> This bug doesn't pop up very often, because HDFS clients such as MapReduce 
> unintentionally calls {{FileSystem#addDelegationTokens}} multiple times.
> We have a custom client that accesses HDFS/KMS-HA using delegation token, and 
> we were puzzled why it always throws "Failed to find any Kerberos tgt" 
> exceptions talking to one KMS but not the other. Turns out that client 
> couldn't talk to the KMS because {{FileSystem#addDelegationTokens}} only gets 
> one KMS delegation token at a time.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-14441) LoadBalancingKMSClientProvider#addDelegationTokens should add delegation tokens from all KMS instances

[Yongjun Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-14441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16021687#comment-16021687
 ] 

Yongjun Zhang commented on HADOOP-14441:


Thanks for working on the issue here guys.

Hi [~shahrs87], it seems your patch fit HADOOP-14445 better. If HADOOP-14445 
works compatibly, we may not need HADOOP-14441. Would you please post your 
patch there even though you are polishing the test now?

Thanks.

> LoadBalancingKMSClientProvider#addDelegationTokens should add delegation 
> tokens from all KMS instances
> --
>
> Key: HADOOP-14441
> URL: https://issues.apache.org/jira/browse/HADOOP-14441
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 2.7.0
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Attachments: HADOOP-14441.001.patch, HADOOP-14441.002.patch, 
> HADOOP-14441.003.patch
>
>
> LoadBalancingKMSClientProvider only gets delegation token from one KMS 
> instance, in a round-robin fashion. This is arguably a bug, as JavaDoc for 
> {{KeyProviderDelegationTokenExtension#addDelegationTokens}} states:
> {quote}
> /**
>  * The implementer of this class will take a renewer and add all
>  * delegation tokens associated with the renewer to the 
>  * Credentials object if it is not already present, 
> ...
> **/
> {quote}
> This bug doesn't pop up very often, because HDFS clients such as MapReduce 
> unintentionally calls {{FileSystem#addDelegationTokens}} multiple times.
> We have a custom client that accesses HDFS/KMS-HA using delegation token, and 
> we were puzzled why it always throws "Failed to find any Kerberos tgt" 
> exceptions talking to one KMS but not the other. Turns out that client 
> couldn't talk to the KMS because {{FileSystem#addDelegationTokens}} only gets 
> one KMS delegation token at a time.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-14441) LoadBalancingKMSClientProvider#addDelegationTokens should add delegation tokens from all KMS instances

[Wei-Chiu Chuang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-14441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16021628#comment-16021628
 ] 

Wei-Chiu Chuang commented on HADOOP-14441:
--

[~shahrs87] thanks for your comments and your effort in creating the patch. 
I've been thinking about alternative way to fix it, but they all turn out to be 
either incompatible (adding extra parameters to public API), or unable to allow 
a client to get delegation tokens from multiple KMS clusters. If your patch is 
incompatible, would you mind to move over to HADOOP-14445 and use this one for 
a short term fix?

Thanks

> LoadBalancingKMSClientProvider#addDelegationTokens should add delegation 
> tokens from all KMS instances
> --
>
> Key: HADOOP-14441
> URL: https://issues.apache.org/jira/browse/HADOOP-14441
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 2.7.0
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Attachments: HADOOP-14441.001.patch, HADOOP-14441.002.patch, 
> HADOOP-14441.003.patch
>
>
> LoadBalancingKMSClientProvider only gets delegation token from one KMS 
> instance, in a round-robin fashion. This is arguably a bug, as JavaDoc for 
> {{KeyProviderDelegationTokenExtension#addDelegationTokens}} states:
> {quote}
> /**
>  * The implementer of this class will take a renewer and add all
>  * delegation tokens associated with the renewer to the 
>  * Credentials object if it is not already present, 
> ...
> **/
> {quote}
> This bug doesn't pop up very often, because HDFS clients such as MapReduce 
> unintentionally calls {{FileSystem#addDelegationTokens}} multiple times.
> We have a custom client that accesses HDFS/KMS-HA using delegation token, and 
> we were puzzled why it always throws "Failed to find any Kerberos tgt" 
> exceptions talking to one KMS but not the other. Turns out that client 
> couldn't talk to the KMS because {{FileSystem#addDelegationTokens}} only gets 
> one KMS delegation token at a time.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-14441) LoadBalancingKMSClientProvider#addDelegationTokens should add delegation tokens from all KMS instances

[Rushabh S Shah (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-14441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16021541#comment-16021541
 ] 

Rushabh S Shah commented on HADOOP-14441:
-

bq. If this helps, RM HA gets around the problem of different host:port for 
different RMs by setting the token's service to host1:port1,host2:port2 (which 
gets stored in ZK and used by both RMs).
Thanks [~rkanter].
That's exactly what I am trying to do.
The test case attached in the patch works on my local machine.
Trying to create a good patch.

> LoadBalancingKMSClientProvider#addDelegationTokens should add delegation 
> tokens from all KMS instances
> --
>
> Key: HADOOP-14441
> URL: https://issues.apache.org/jira/browse/HADOOP-14441
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 2.7.0
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Attachments: HADOOP-14441.001.patch, HADOOP-14441.002.patch, 
> HADOOP-14441.003.patch
>
>
> LoadBalancingKMSClientProvider only gets delegation token from one KMS 
> instance, in a round-robin fashion. This is arguably a bug, as JavaDoc for 
> {{KeyProviderDelegationTokenExtension#addDelegationTokens}} states:
> {quote}
> /**
>  * The implementer of this class will take a renewer and add all
>  * delegation tokens associated with the renewer to the 
>  * Credentials object if it is not already present, 
> ...
> **/
> {quote}
> This bug doesn't pop up very often, because HDFS clients such as MapReduce 
> unintentionally calls {{FileSystem#addDelegationTokens}} multiple times.
> We have a custom client that accesses HDFS/KMS-HA using delegation token, and 
> we were puzzled why it always throws "Failed to find any Kerberos tgt" 
> exceptions talking to one KMS but not the other. Turns out that client 
> couldn't talk to the KMS because {{FileSystem#addDelegationTokens}} only gets 
> one KMS delegation token at a time.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-14441) LoadBalancingKMSClientProvider#addDelegationTokens should add delegation tokens from all KMS instances

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-14441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16021522#comment-16021522
 ] 

Robert Kanter commented on HADOOP-14441:


If this helps, RM HA gets around the problem of different host:port for 
different RMs by setting the token's service to {{host1:port1,host2:port2}} 
(which gets stored in ZK and used by both RMs).
https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/ClientRMProxy.java#L144

> LoadBalancingKMSClientProvider#addDelegationTokens should add delegation 
> tokens from all KMS instances
> --
>
> Key: HADOOP-14441
> URL: https://issues.apache.org/jira/browse/HADOOP-14441
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 2.7.0
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Attachments: HADOOP-14441.001.patch, HADOOP-14441.002.patch, 
> HADOOP-14441.003.patch
>
>
> LoadBalancingKMSClientProvider only gets delegation token from one KMS 
> instance, in a round-robin fashion. This is arguably a bug, as JavaDoc for 
> {{KeyProviderDelegationTokenExtension#addDelegationTokens}} states:
> {quote}
> /**
>  * The implementer of this class will take a renewer and add all
>  * delegation tokens associated with the renewer to the 
>  * Credentials object if it is not already present, 
> ...
> **/
> {quote}
> This bug doesn't pop up very often, because HDFS clients such as MapReduce 
> unintentionally calls {{FileSystem#addDelegationTokens}} multiple times.
> We have a custom client that accesses HDFS/KMS-HA using delegation token, and 
> we were puzzled why it always throws "Failed to find any Kerberos tgt" 
> exceptions talking to one KMS but not the other. Turns out that client 
> couldn't talk to the KMS because {{FileSystem#addDelegationTokens}} only gets 
> one KMS delegation token at a time.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-14441) LoadBalancingKMSClientProvider#addDelegationTokens should add delegation tokens from all KMS instances

[Wei-Chiu Chuang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-14441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16020451#comment-16020451
 ] 

Wei-Chiu Chuang commented on HADOOP-14441:
--

You can take a look at the test in my patch. I had to workaround that 
limitation as well. (See: createHAProvider())

> LoadBalancingKMSClientProvider#addDelegationTokens should add delegation 
> tokens from all KMS instances
> --
>
> Key: HADOOP-14441
> URL: https://issues.apache.org/jira/browse/HADOOP-14441
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 2.7.0
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Attachments: HADOOP-14441.001.patch, HADOOP-14441.002.patch, 
> HADOOP-14441.003.patch
>
>
> LoadBalancingKMSClientProvider only gets delegation token from one KMS 
> instance, in a round-robin fashion. This is arguably a bug, as JavaDoc for 
> {{KeyProviderDelegationTokenExtension#addDelegationTokens}} states:
> {quote}
> /**
>  * The implementer of this class will take a renewer and add all
>  * delegation tokens associated with the renewer to the 
>  * Credentials object if it is not already present, 
> ...
> **/
> {quote}
> This bug doesn't pop up very often, because HDFS clients such as MapReduce 
> unintentionally calls {{FileSystem#addDelegationTokens}} multiple times.
> We have a custom client that accesses HDFS/KMS-HA using delegation token, and 
> we were puzzled why it always throws "Failed to find any Kerberos tgt" 
> exceptions talking to one KMS but not the other. Turns out that client 
> couldn't talk to the KMS because {{FileSystem#addDelegationTokens}} only gets 
> one KMS delegation token at a time.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-14441) LoadBalancingKMSClientProvider#addDelegationTokens should add delegation tokens from all KMS instances

[Arun Suresh (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-14441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16020375#comment-16020375
 ] 

Arun Suresh commented on HADOOP-14441:
--

bq. Does anyone know how to create LBKMSClientProviderUrl with different ports ?
Using the default factory, you can't. The way around this is to create a dummy 
KeyProviderFactory that extends the KMSKeyProviderFactory that constructs the 
LBKMSClientProvider the way your test case desires. The name of the Dummy Key 
Provider should then be added to the 
resources/META-INF/services/org.apache.hadoop.crypto.key.KeyProviderFactory 
file so that it can be instantiated.

> LoadBalancingKMSClientProvider#addDelegationTokens should add delegation 
> tokens from all KMS instances
> --
>
> Key: HADOOP-14441
> URL: https://issues.apache.org/jira/browse/HADOOP-14441
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 2.7.0
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Attachments: HADOOP-14441.001.patch, HADOOP-14441.002.patch, 
> HADOOP-14441.003.patch
>
>
> LoadBalancingKMSClientProvider only gets delegation token from one KMS 
> instance, in a round-robin fashion. This is arguably a bug, as JavaDoc for 
> {{KeyProviderDelegationTokenExtension#addDelegationTokens}} states:
> {quote}
> /**
>  * The implementer of this class will take a renewer and add all
>  * delegation tokens associated with the renewer to the 
>  * Credentials object if it is not already present, 
> ...
> **/
> {quote}
> This bug doesn't pop up very often, because HDFS clients such as MapReduce 
> unintentionally calls {{FileSystem#addDelegationTokens}} multiple times.
> We have a custom client that accesses HDFS/KMS-HA using delegation token, and 
> we were puzzled why it always throws "Failed to find any Kerberos tgt" 
> exceptions talking to one KMS but not the other. Turns out that client 
> couldn't talk to the KMS because {{FileSystem#addDelegationTokens}} only gets 
> one KMS delegation token at a time.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-14441) LoadBalancingKMSClientProvider#addDelegationTokens should add delegation tokens from all KMS instances

[Arun Suresh (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-14441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16020356#comment-16020356
 ] 

Arun Suresh commented on HADOOP-14441:
--

[~shahrs87], have you tried this by any chance ?
bq. First time we get a DT from any one of the kms instances, we store the same 
DT against ALL the service urls in the user credential. This would require the 
ZKDTSM to be configured, to replicate the DT to all kms instances.
And yeah, the LBKMSClientProvider requires all services to use the same port.

> LoadBalancingKMSClientProvider#addDelegationTokens should add delegation 
> tokens from all KMS instances
> --
>
> Key: HADOOP-14441
> URL: https://issues.apache.org/jira/browse/HADOOP-14441
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 2.7.0
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Attachments: HADOOP-14441.001.patch, HADOOP-14441.002.patch, 
> HADOOP-14441.003.patch
>
>
> LoadBalancingKMSClientProvider only gets delegation token from one KMS 
> instance, in a round-robin fashion. This is arguably a bug, as JavaDoc for 
> {{KeyProviderDelegationTokenExtension#addDelegationTokens}} states:
> {quote}
> /**
>  * The implementer of this class will take a renewer and add all
>  * delegation tokens associated with the renewer to the 
>  * Credentials object if it is not already present, 
> ...
> **/
> {quote}
> This bug doesn't pop up very often, because HDFS clients such as MapReduce 
> unintentionally calls {{FileSystem#addDelegationTokens}} multiple times.
> We have a custom client that accesses HDFS/KMS-HA using delegation token, and 
> we were puzzled why it always throws "Failed to find any Kerberos tgt" 
> exceptions talking to one KMS but not the other. Turns out that client 
> couldn't talk to the KMS because {{FileSystem#addDelegationTokens}} only gets 
> one KMS delegation token at a time.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-14441) LoadBalancingKMSClientProvider#addDelegationTokens should add delegation tokens from all KMS instances

[Rushabh S Shah (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-14441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16020339#comment-16020339
 ] 

Rushabh S Shah commented on HADOOP-14441:
-

I sort of have a sample patch ready which will not need getting delegation 
token from all the providers.
While writing the test case, I found that there is no way to create 
LoadBalancingKmsClientProvider uri with different port for each provider.
Since unit tests will always have hostname as localhost, the only 
differentiating factor would be the port number.
In the source code there is comment which says the same thing.
{code:title=KMSClientProvider.java|borderStyle=solid}
 public KeyProvider createProvider(URI providerUri, Configuration conf)
{


  // Check if port is present in authority
// In the current scheme, all hosts have to run on the same port
int port = -1;
String hostsPart = authority;
if (authority.contains(":")) {
  String[] t = authority.split(":");
  try {
port = Integer.parseInt(t[1]);
  } catch (Exception e) {
throw new IOException(
"Could not parse port in kms uri [" + origUrl + "]");
  }
  hostsPart = t[0];
}
return createProvider(providerUri, conf, origUrl, port, hostsPart);
}
{code}
Does anyone know how to create LBKMSClientProviderUrl with different ports ?

> LoadBalancingKMSClientProvider#addDelegationTokens should add delegation 
> tokens from all KMS instances
> --
>
> Key: HADOOP-14441
> URL: https://issues.apache.org/jira/browse/HADOOP-14441
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 2.7.0
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Attachments: HADOOP-14441.001.patch, HADOOP-14441.002.patch, 
> HADOOP-14441.003.patch
>
>
> LoadBalancingKMSClientProvider only gets delegation token from one KMS 
> instance, in a round-robin fashion. This is arguably a bug, as JavaDoc for 
> {{KeyProviderDelegationTokenExtension#addDelegationTokens}} states:
> {quote}
> /**
>  * The implementer of this class will take a renewer and add all
>  * delegation tokens associated with the renewer to the 
>  * Credentials object if it is not already present, 
> ...
> **/
> {quote}
> This bug doesn't pop up very often, because HDFS clients such as MapReduce 
> unintentionally calls {{FileSystem#addDelegationTokens}} multiple times.
> We have a custom client that accesses HDFS/KMS-HA using delegation token, and 
> we were puzzled why it always throws "Failed to find any Kerberos tgt" 
> exceptions talking to one KMS but not the other. Turns out that client 
> couldn't talk to the KMS because {{FileSystem#addDelegationTokens}} only gets 
> one KMS delegation token at a time.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-14441) LoadBalancingKMSClientProvider#addDelegationTokens should add delegation tokens from all KMS instances

[Wei-Chiu Chuang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-14441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16020165#comment-16020165
 ] 

Wei-Chiu Chuang commented on HADOOP-14441:
--

Sure. Thanks

> LoadBalancingKMSClientProvider#addDelegationTokens should add delegation 
> tokens from all KMS instances
> --
>
> Key: HADOOP-14441
> URL: https://issues.apache.org/jira/browse/HADOOP-14441
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 2.7.0
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Attachments: HADOOP-14441.001.patch, HADOOP-14441.002.patch, 
> HADOOP-14441.003.patch
>
>
> LoadBalancingKMSClientProvider only gets delegation token from one KMS 
> instance, in a round-robin fashion. This is arguably a bug, as JavaDoc for 
> {{KeyProviderDelegationTokenExtension#addDelegationTokens}} states:
> {quote}
> /**
>  * The implementer of this class will take a renewer and add all
>  * delegation tokens associated with the renewer to the 
>  * Credentials object if it is not already present, 
> ...
> **/
> {quote}
> This bug doesn't pop up very often, because HDFS clients such as MapReduce 
> unintentionally calls {{FileSystem#addDelegationTokens}} multiple times.
> We have a custom client that accesses HDFS/KMS-HA using delegation token, and 
> we were puzzled why it always throws "Failed to find any Kerberos tgt" 
> exceptions talking to one KMS but not the other. Turns out that client 
> couldn't talk to the KMS because {{FileSystem#addDelegationTokens}} only gets 
> one KMS delegation token at a time.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-14441) LoadBalancingKMSClientProvider#addDelegationTokens should add delegation tokens from all KMS instances

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-14441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16019980#comment-16019980
 ] 

Hadoop QA commented on HADOOP-14441:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
31s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
22s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m 
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m  
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
42s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  1m 
33s{color} | {color:red} hadoop-common-project/hadoop-common in trunk has 19 
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
9s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
 0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m 
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 13m 
23s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 43s{color} | {color:orange} hadoop-common-project: The patch generated 4 new 
+ 101 unchanged - 3 fixed = 105 total (was 104) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  1m 
42s{color} | {color:red} hadoop-common-project/hadoop-common generated 1 new + 
19 unchanged - 0 fixed = 20 total (was 19) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  8m 
18s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  3m  
8s{color} | {color:green} hadoop-kms in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
37s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 75m 40s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-common-project/hadoop-common |
|  |  instanceof will always return true for all non-null values in 
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.addDelegationTokens(String,
 Credentials), since all RuntimeException are instances of RuntimeException  At 
LoadBalancingKMSClientProvider.java:for all non-null values in 
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.addDelegationTokens(String,
 Credentials), since all RuntimeException are instances of RuntimeException  At 
LoadBalancingKMSClientProvider.java:[line 153] |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HADOOP-14441 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12869273/HADOOP-14441.003.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux edab20cf5ac6 3.13.0-116-generic #163-Ubuntu SMP Fri 

[jira] [Commented] (HADOOP-14441) LoadBalancingKMSClientProvider#addDelegationTokens should add delegation tokens from all KMS instances

[Rushabh S Shah (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-14441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16019938#comment-16019938
 ] 

Rushabh S Shah commented on HADOOP-14441:
-

Can we wait till EOD before checking in this patch ?

> LoadBalancingKMSClientProvider#addDelegationTokens should add delegation 
> tokens from all KMS instances
> --
>
> Key: HADOOP-14441
> URL: https://issues.apache.org/jira/browse/HADOOP-14441
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 2.7.0
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Attachments: HADOOP-14441.001.patch, HADOOP-14441.002.patch, 
> HADOOP-14441.003.patch
>
>
> LoadBalancingKMSClientProvider only gets delegation token from one KMS 
> instance, in a round-robin fashion. This is arguably a bug, as JavaDoc for 
> {{KeyProviderDelegationTokenExtension#addDelegationTokens}} states:
> {quote}
> /**
>  * The implementer of this class will take a renewer and add all
>  * delegation tokens associated with the renewer to the 
>  * Credentials object if it is not already present, 
> ...
> **/
> {quote}
> This bug doesn't pop up very often, because HDFS clients such as MapReduce 
> unintentionally calls {{FileSystem#addDelegationTokens}} multiple times.
> We have a custom client that accesses HDFS/KMS-HA using delegation token, and 
> we were puzzled why it always throws "Failed to find any Kerberos tgt" 
> exceptions talking to one KMS but not the other. Turns out that client 
> couldn't talk to the KMS because {{FileSystem#addDelegationTokens}} only gets 
> one KMS delegation token at a time.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-14441) LoadBalancingKMSClientProvider#addDelegationTokens should add delegation tokens from all KMS instances

[Rushabh S Shah (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-14441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16019925#comment-16019925
 ] 

Rushabh S Shah commented on HADOOP-14441:
-

I agree this is a problem.
Right now we can fix it temporarily but getting delegation tokens from all the 
providers would pose problems at scale.
We need to rethink for a scalable solution later on.



> LoadBalancingKMSClientProvider#addDelegationTokens should add delegation 
> tokens from all KMS instances
> --
>
> Key: HADOOP-14441
> URL: https://issues.apache.org/jira/browse/HADOOP-14441
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 2.7.0
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Attachments: HADOOP-14441.001.patch, HADOOP-14441.002.patch, 
> HADOOP-14441.003.patch
>
>
> LoadBalancingKMSClientProvider only gets delegation token from one KMS 
> instance, in a round-robin fashion. This is arguably a bug, as JavaDoc for 
> {{KeyProviderDelegationTokenExtension#addDelegationTokens}} states:
> {quote}
> /**
>  * The implementer of this class will take a renewer and add all
>  * delegation tokens associated with the renewer to the 
>  * Credentials object if it is not already present, 
> ...
> **/
> {quote}
> This bug doesn't pop up very often, because HDFS clients such as MapReduce 
> unintentionally calls {{FileSystem#addDelegationTokens}} multiple times.
> We have a custom client that accesses HDFS/KMS-HA using delegation token, and 
> we were puzzled why it always throws "Failed to find any Kerberos tgt" 
> exceptions talking to one KMS but not the other. Turns out that client 
> couldn't talk to the KMS because {{FileSystem#addDelegationTokens}} only gets 
> one KMS delegation token at a time.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-14441) LoadBalancingKMSClientProvider#addDelegationTokens should add delegation tokens from all KMS instances

[Arun Suresh (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-14441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16019893#comment-16019893
 ] 

Arun Suresh commented on HADOOP-14441:
--

Thanks for the clarification [~jojochuang]..
I see your point. So I guess there are possibly 2 ways to fix this:
# As you suggested, perhaps have the LBKMSProvider collect delegation tokens 
from EACH kms instance and store it in the client credential against it 
corresponding service url. This would mean we might not need to use ZKDTSM to 
replicate the DTs across all KMS instances.
# First time we get a DT from any one of the kms instances, we store the same 
DT against ALL the service urls in the user credential. This would require the 
ZKDTSM to be configured, to replicate the DT to all kms instances.
Not sure about how involved the changes for option 2 would be though.


> LoadBalancingKMSClientProvider#addDelegationTokens should add delegation 
> tokens from all KMS instances
> --
>
> Key: HADOOP-14441
> URL: https://issues.apache.org/jira/browse/HADOOP-14441
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 2.7.0
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Attachments: HADOOP-14441.001.patch, HADOOP-14441.002.patch, 
> HADOOP-14441.003.patch
>
>
> LoadBalancingKMSClientProvider only gets delegation token from one KMS 
> instance, in a round-robin fashion. This is arguably a bug, as JavaDoc for 
> {{KeyProviderDelegationTokenExtension#addDelegationTokens}} states:
> {quote}
> /**
>  * The implementer of this class will take a renewer and add all
>  * delegation tokens associated with the renewer to the 
>  * Credentials object if it is not already present, 
> ...
> **/
> {quote}
> This bug doesn't pop up very often, because HDFS clients such as MapReduce 
> unintentionally calls {{FileSystem#addDelegationTokens}} multiple times.
> We have a custom client that accesses HDFS/KMS-HA using delegation token, and 
> we were puzzled why it always throws "Failed to find any Kerberos tgt" 
> exceptions talking to one KMS but not the other. Turns out that client 
> couldn't talk to the KMS because {{FileSystem#addDelegationTokens}} only gets 
> one KMS delegation token at a time.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-14441) LoadBalancingKMSClientProvider#addDelegationTokens should add delegation tokens from all KMS instances

[Wei-Chiu Chuang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-14441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16019857#comment-16019857
 ] 

Wei-Chiu Chuang commented on HADOOP-14441:
--

Hi [~asuresh] thanks for commenting on this.
bq. I agree with Rushabh S Shah. If the KMS is configured to use the ZK 
delegationtoken secret manager, you would not need to add the DTs from all KMS 
instances.
My test uses doKMSWithZK() which has 
hadoop.kms.authentication.zk-dt-secret-manager.enable = true.
bq. Since a DT issued by 1 KMS instance will be validated by any of its peers.
I agree with you completely.
The problem is the client side does not know that it should authenticate the 
KMS#2 using the delegation token obtained from KMS#1. Please take a look at the 
following code:

{code:title=DelegationTokenAuthenticatedURL#openConnection}
// delegation token
  Credentials creds = UserGroupInformation.getCurrentUser().
  getCredentials();
  if (!creds.getAllTokens().isEmpty()) {
InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
url.getPort());
Text service = SecurityUtil.buildTokenService(serviceAddr);
dToken = creds.getToken(service); <- this becomes null in my test 
case.
if (dToken != null) {
  if (useQueryStringForDelegationToken()) {
// delegation token will go in the query string, injecting it
extraParams.put(
KerberosDelegationTokenAuthenticator.DELEGATION_PARAM,
dToken.encodeToUrlString());
  } else {
// delegation token will go as request header, setting it in the
// auth-token to ensure no authentication handshake is triggered
// (if we have a delegation token, we are authenticated)
// the delegation token header is injected in the connection request
// at the end of this method.
token.delegationToken = (org.apache.hadoop.security.token.Token
) dToken;
  }
}
  }
{code}

> LoadBalancingKMSClientProvider#addDelegationTokens should add delegation 
> tokens from all KMS instances
> --
>
> Key: HADOOP-14441
> URL: https://issues.apache.org/jira/browse/HADOOP-14441
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 2.7.0
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Attachments: HADOOP-14441.001.patch, HADOOP-14441.002.patch, 
> HADOOP-14441.003.patch
>
>
> LoadBalancingKMSClientProvider only gets delegation token from one KMS 
> instance, in a round-robin fashion. This is arguably a bug, as JavaDoc for 
> {{KeyProviderDelegationTokenExtension#addDelegationTokens}} states:
> {quote}
> /**
>  * The implementer of this class will take a renewer and add all
>  * delegation tokens associated with the renewer to the 
>  * Credentials object if it is not already present, 
> ...
> **/
> {quote}
> This bug doesn't pop up very often, because HDFS clients such as MapReduce 
> unintentionally calls {{FileSystem#addDelegationTokens}} multiple times.
> We have a custom client that accesses HDFS/KMS-HA using delegation token, and 
> we were puzzled why it always throws "Failed to find any Kerberos tgt" 
> exceptions talking to one KMS but not the other. Turns out that client 
> couldn't talk to the KMS because {{FileSystem#addDelegationTokens}} only gets 
> one KMS delegation token at a time.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-14441) LoadBalancingKMSClientProvider#addDelegationTokens should add delegation tokens from all KMS instances

[Wei-Chiu Chuang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-14441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16019838#comment-16019838
 ] 

Wei-Chiu Chuang commented on HADOOP-14441:
--

Here's some sample stack trace. Note I added additional debug logs so it's 
quite cluttered.

2017-05-17 18:44:53,382 DEBUG LoadBalancingKMSClientProvider - trying provider 
https://weichiu-foo-3.example.com:16000/kms/v1/
2017-05-17 18:44:53,383 WARN  Token - Cannot find class for token kind kms-dt
2017-05-17 18:44:53,384 WARN  Token - Cannot find class for token kind kms-dt
2017-05-17 18:44:53,384 DEBUG KMSClientProvider - KMS provider 
[https://weichiu-foo-3.example.com:16000/kms/v1/] actual ugi = foo 
(auth:KERBEROS) subject=Subject:
Principal: UnixPrincipal: foo
Principal: UnixNumericUserPrincipal: 2004
Principal: UnixNumericGroupPrincipal [Primary Group]: 2004
Principal: foo
Private Credential: tokenMap: key=172.31.117.206:8032 value=Kind: 
RM_DELEGATION_TOKEN, Service: 172.31.117.206:8032, Ident: 00 18 61 74 74 69 76 
69 6f 40 47 43 45 2e 43 4c 4f
55 44 45 52 41 2e 43 4f 4d 04 79 61 72 6e 00 8a 01 5c 19 39 a4 55 8a 01 5c 3d 
46 28 55 1a 02;
key=ha-hdfs:ns1 value=Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:ns1, Ident: 
(HDFS_DELEGATION_TOKEN token 110 for foo);
key=172.31.123.173:16000 value=Kind: kms-dt, Service: 172.31.123.173:16000, 
Ident: 00 07 61 74 74 69 76 69 6f 04 79 61 72 6e 00 8a 01 5c 19 39 a4 43 8a 01 
5c 3d 46 28 43 25 22;
secretKeysMap:
 current ugi=foo (auth:KERBEROS) subject=Subject:
Principal: UnixPrincipal: foo
Principal: UnixNumericUserPrincipal: 2004
Principal: UnixNumericGroupPrincipal [Primary Group]: 2004
Principal: foo
Private Credential: tokenMap: key=172.31.117.206:8032 value=Kind: 
RM_DELEGATION_TOKEN, Service: 172.31.117.206:8032, Ident: 00 18 61 74 74 69 76 
69 6f 40 47 43 45 2e 43 4c 4f 55 44 45 52 41 2e 43 4f 4d 04 79 61 72 6e 00 8a 
01 5c 19 39 a4 55 8a 01 5c 3d 46 28 55 1a 02;
key=ha-hdfs:ns1 value=Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:ns1, Ident: 
(HDFS_DELEGATION_TOKEN token 110 for foo);
key=172.31.123.173:16000 value=Kind: kms-dt, Service: 172.31.123.173:16000, 
Ident: 00 07 61 74 74 69 76 69 6f 04 79 61 72 6e 00 8a 01 5c 19 39 a4 43 8a 01 
5c 3d 46 28 43 25 22;
secretKeysMap:
 
url=https://weichiu-foo-3.example.com:16000/kms/v1/keyversion/yH32H7e2tnhd38HGrb45OlrG4xHYJheOs4ITA5NhZbr/_eek?eek_op=decrypt
 authToken=null doAsUser=null
2017-05-17 18:44:53,385 WARN  Token - Cannot find class for token kind kms-dt
2017-05-17 18:44:53,386 WARN  Token - Cannot find class for token kind kms-dt
2017-05-17 18:44:53,388 DEBUG UserGroupInformation - PrivilegedAction as:foo 
(auth:KERBEROS) subject=Subject:
Principal: UnixPrincipal: foo
Principal: UnixNumericUserPrincipal: 2004
Principal: UnixNumericGroupPrincipal [Primary Group]: 2004
Principal: foo
Private Credential: tokenMap: key=172.31.117.206:8032 value=Kind: 
RM_DELEGATION_TOKEN, Service: 172.31.117.206:8032, Ident: 00 18 61 74 74 69 76 
69 6f 40 47 43 45 2e 43 4c 4f 55 44 45 52 41 2e 43 4f 4d 04 79 61 72 6e 00 8a 
01 5c 19 39 a4 55 8a 01 5c 3d 46 28 55 1a 02;
key=ha-hdfs:ns1 value=Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:ns1, Ident: 
(HDFS_DELEGATION_TOKEN token 110 for foo);
key=172.31.123.173:16000 value=Kind: kms-dt, Service: 172.31.123.173:16000, 
Ident: 00 07 61 74 74 69 76 69 6f 04 79 61 72 6e 00 8a 01 5c 19 39 a4 43 8a 01 
5c 3d 46 28 43 25 22;
secretKeysMap:
 
from:org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:489).
 subject=Subject:
Principal: UnixPrincipal: foo
Principal: UnixNumericUserPrincipal: 2004
Principal: UnixNumericGroupPrincipal [Primary Group]: 2004
Principal: foo
Private Credential: tokenMap: key=172.31.117.206:8032 value=Kind: 
RM_DELEGATION_TOKEN, Service: 172.31.117.206:8032, Ident: 00 18 61 74 74 69 76 
69 6f 40 47 43 45 2e 43 4c 4f 55 44 45 52 41 2e 43 4f 4d 04 79 61 72 6e 00 8a 
01 5c 19 39 a4 55 8a 01 5c 3d 46 28 55 1a 02;
key=ha-hdfs:ns1 value=Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:ns1, Ident: 
(HDFS_DELEGATION_TOKEN token 110 for foo);
key=172.31.123.173:16000 value=Kind: kms-dt, Service: 172.31.123.173:16000, 
Ident: 00 07 61 74 74 69 76 69 6f 04 79 61 72 6e 00 8a 01 5c 19 39 a4 43 8a 01 
5c 3d 46 28 43 25 22;
secretKeysMap:
.java.lang.Throwable
at 
org.apache.hadoop.security.UserGroupInformation.logPrivilegedAction(UserGroupInformation.java:1687)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1662)
at 
org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:489)
at 
org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:787)
at 

[jira] [Commented] (HADOOP-14441) LoadBalancingKMSClientProvider#addDelegationTokens should add delegation tokens from all KMS instances

[Arun Suresh (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-14441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16019837#comment-16019837
 ] 

Arun Suresh commented on HADOOP-14441:
--

I agree with [~shahrs87]. If the KMS is configured to use the ZK 
delegationtoken secret manager, you would not to add the DTs from all KMS 
instances.
Since a DT issued by 1 KMS instance will be validated by any of its peers.

> LoadBalancingKMSClientProvider#addDelegationTokens should add delegation 
> tokens from all KMS instances
> --
>
> Key: HADOOP-14441
> URL: https://issues.apache.org/jira/browse/HADOOP-14441
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 2.7.0
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Attachments: HADOOP-14441.001.patch, HADOOP-14441.002.patch, 
> HADOOP-14441.003.patch
>
>
> LoadBalancingKMSClientProvider only gets delegation token from one KMS 
> instance, in a round-robin fashion. This is arguably a bug, as JavaDoc for 
> {{KeyProviderDelegationTokenExtension#addDelegationTokens}} states:
> {quote}
> /**
>  * The implementer of this class will take a renewer and add all
>  * delegation tokens associated with the renewer to the 
>  * Credentials object if it is not already present, 
> ...
> **/
> {quote}
> This bug doesn't pop up very often, because HDFS clients such as MapReduce 
> unintentionally calls {{FileSystem#addDelegationTokens}} multiple times.
> We have a custom client that accesses HDFS/KMS-HA using delegation token, and 
> we were puzzled why it always throws "Failed to find any Kerberos tgt" 
> exceptions talking to one KMS but not the other. Turns out that client 
> couldn't talk to the KMS because {{FileSystem#addDelegationTokens}} only gets 
> one KMS delegation token at a time.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-14441) LoadBalancingKMSClientProvider#addDelegationTokens should add delegation tokens from all KMS instances

[Wei-Chiu Chuang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-14441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16019834#comment-16019834
 ] 

Wei-Chiu Chuang commented on HADOOP-14441:
--

Hi [~shahrs87], There are two ways to configure KMS-HA: one is to use KMS 
servers behind VIP, and the other is via LoadBalancingKMSClientProvider, which 
is adopted by Cloudera. From a high level perspective, clients are not aware of 
KMS HA in the former configuration, and the VIP is responsible for routing the 
requests; while in the latter, the client are aware there are multiple KMS 
servers and itself is responsible for routing the requests to the KMS servers.

The bug described here is purely a problem using LoadBalancingKMSClientProvider 
configuration. When a KMS client requests a delegation token from KMS server, 
it uses the server address/port as the key to store dt in its UGI Credentials 
map:

{code:title=DelegationTokenAuthenticatedURL#getDelegationToken}
public org.apache.hadoop.security.token.Token
  getDelegationToken(URL url, Token token, String renewer, String doAsUser)
  throws IOException, AuthenticationException {
Preconditions.checkNotNull(url, "url");
Preconditions.checkNotNull(token, "token");
try {
  token.delegationToken =
  ((KerberosDelegationTokenAuthenticator) getAuthenticator()).
  getDelegationToken(url, token, renewer, doAsUser);
  return token.delegationToken;
} catch (IOException ex) {
  token.delegationToken = null;
  throw ex;
}
  }
{code}
The problem is that the client is aware of the real server addreess/port, so 
when it looks up its Credentials map, the delegation token acquired from one 
KMS server can not be used for another KMS server.

The test case attached to this jira accurately capture the problem and the 
error.

bq. Even after the fix, the jobs can fail if one the servers went temporarily 
down and came back later and if the job was launched in between these time 
frame.
I agree this is a problem. Presumably there's a way for KMS to share the same 
URL, but the current Hadoop Authentication framework is shared by multiple 
agents including YARN client, so I am not sure what would be a better approach 
to fix it without affect other agents.

> LoadBalancingKMSClientProvider#addDelegationTokens should add delegation 
> tokens from all KMS instances
> --
>
> Key: HADOOP-14441
> URL: https://issues.apache.org/jira/browse/HADOOP-14441
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 2.7.0
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Attachments: HADOOP-14441.001.patch, HADOOP-14441.002.patch, 
> HADOOP-14441.003.patch
>
>
> LoadBalancingKMSClientProvider only gets delegation token from one KMS 
> instance, in a round-robin fashion. This is arguably a bug, as JavaDoc for 
> {{KeyProviderDelegationTokenExtension#addDelegationTokens}} states:
> {quote}
> /**
>  * The implementer of this class will take a renewer and add all
>  * delegation tokens associated with the renewer to the 
>  * Credentials object if it is not already present, 
> ...
> **/
> {quote}
> This bug doesn't pop up very often, because HDFS clients such as MapReduce 
> unintentionally calls {{FileSystem#addDelegationTokens}} multiple times.
> We have a custom client that accesses HDFS/KMS-HA using delegation token, and 
> we were puzzled why it always throws "Failed to find any Kerberos tgt" 
> exceptions talking to one KMS but not the other. Turns out that client 
> couldn't talk to the KMS because {{FileSystem#addDelegationTokens}} only gets 
> one KMS delegation token at a time.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-14441) LoadBalancingKMSClientProvider#addDelegationTokens should add delegation tokens from all KMS instances

[Rushabh S Shah (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-14441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16019786#comment-16019786
 ] 

Rushabh S Shah commented on HADOOP-14441:
-

[~jojochuang] We run sort of kms-ha with having 2 servers behind vip.
We don't face this issue. We have back-end zookeeper to sync the state between 
multiple kms servers.
Even if you have zookeeper enabled, what is the value of this config 
{{hadoop.kms.authentication.zk-dt-secret-manager.enable}} ?
Can you post some stack trace ?
I don't think the correct fix is to get delegation token from all the servers.
Even after the fix, the jobs can fail if one the servers went temporarily down 
and came back later and if the job was launched in between these time frame.

> LoadBalancingKMSClientProvider#addDelegationTokens should add delegation 
> tokens from all KMS instances
> --
>
> Key: HADOOP-14441
> URL: https://issues.apache.org/jira/browse/HADOOP-14441
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 2.7.0
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Attachments: HADOOP-14441.001.patch, HADOOP-14441.002.patch, 
> HADOOP-14441.003.patch
>
>
> LoadBalancingKMSClientProvider only gets delegation token from one KMS 
> instance, in a round-robin fashion. This is arguably a bug, as JavaDoc for 
> {{KeyProviderDelegationTokenExtension#addDelegationTokens}} states:
> {quote}
> /**
>  * The implementer of this class will take a renewer and add all
>  * delegation tokens associated with the renewer to the 
>  * Credentials object if it is not already present, 
> ...
> **/
> {quote}
> This bug doesn't pop up very often, because HDFS clients such as MapReduce 
> unintentionally calls {{FileSystem#addDelegationTokens}} multiple times.
> We have a custom client that accesses HDFS/KMS-HA using delegation token, and 
> we were puzzled why it always throws "Failed to find any Kerberos tgt" 
> exceptions talking to one KMS but not the other. Turns out that client 
> couldn't talk to the KMS because {{FileSystem#addDelegationTokens}} only gets 
> one KMS delegation token at a time.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-14441) LoadBalancingKMSClientProvider#addDelegationTokens should add delegation tokens from all KMS instances

[Yongjun Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-14441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16018228#comment-16018228
 ] 

Yongjun Zhang commented on HADOOP-14441:


Hi [~jojochuang], 

The fix side looks good to me. I'd suggest to add some comment in the test 
code, to explain/indicate the expected outcome of the different parts in 
doKMSHAWithZKWithDelegationToken.

Thanks.


> LoadBalancingKMSClientProvider#addDelegationTokens should add delegation 
> tokens from all KMS instances
> --
>
> Key: HADOOP-14441
> URL: https://issues.apache.org/jira/browse/HADOOP-14441
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 2.7.0
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Attachments: HADOOP-14441.001.patch, HADOOP-14441.002.patch
>
>
> LoadBalancingKMSClientProvider only gets delegation token from one KMS 
> instance, in a round-robin fashion. This is arguably a bug, as JavaDoc for 
> {{KeyProviderDelegationTokenExtension#addDelegationTokens}} states:
> {quote}
> /**
>  * The implementer of this class will take a renewer and add all
>  * delegation tokens associated with the renewer to the 
>  * Credentials object if it is not already present, 
> ...
> **/
> {quote}
> This bug doesn't pop up very often, because HDFS clients such as MapReduce 
> unintentionally calls {{FileSystem#addDelegationTokens}} multiple times.
> We have a custom client that accesses HDFS/KMS-HA using delegation token, and 
> we were puzzled why it always throws "Failed to find any Kerberos tgt" 
> exceptions talking to one KMS but not the other. Turns out that client 
> couldn't talk to the KMS because {{FileSystem#addDelegationTokens}} only gets 
> one KMS delegation token at a time.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-14441) LoadBalancingKMSClientProvider#addDelegationTokens should add delegation tokens from all KMS instances

[Yongjun Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-14441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16018070#comment-16018070
 ] 

Yongjun Zhang commented on HADOOP-14441:


Oops sorry, somehow I missed that. Thanks!


> LoadBalancingKMSClientProvider#addDelegationTokens should add delegation 
> tokens from all KMS instances
> --
>
> Key: HADOOP-14441
> URL: https://issues.apache.org/jira/browse/HADOOP-14441
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 2.7.0
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Attachments: HADOOP-14441.001.patch, HADOOP-14441.002.patch
>
>
> LoadBalancingKMSClientProvider only gets delegation token from one KMS 
> instance, in a round-robin fashion. This is arguably a bug, as JavaDoc for 
> {{KeyProviderDelegationTokenExtension#addDelegationTokens}} states:
> {quote}
> /**
>  * The implementer of this class will take a renewer and add all
>  * delegation tokens associated with the renewer to the 
>  * Credentials object if it is not already present, 
> ...
> **/
> {quote}
> This bug doesn't pop up very often, because HDFS clients such as MapReduce 
> unintentionally calls {{FileSystem#addDelegationTokens}} multiple times.
> We have a custom client that accesses HDFS/KMS-HA using delegation token, and 
> we were puzzled why it always throws "Failed to find any Kerberos tgt" 
> exceptions talking to one KMS but not the other. Turns out that client 
> couldn't talk to the KMS because {{FileSystem#addDelegationTokens}} only gets 
> one KMS delegation token at a time.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-14441) LoadBalancingKMSClientProvider#addDelegationTokens should add delegation tokens from all KMS instances

[Wei-Chiu Chuang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-14441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16018050#comment-16018050
 ] 

Wei-Chiu Chuang commented on HADOOP-14441:
--

the doOp() has a for loop. If one KMS provider fails for the request, the 
exception is caught and it goes to the next one, until all of them are tried.

> LoadBalancingKMSClientProvider#addDelegationTokens should add delegation 
> tokens from all KMS instances
> --
>
> Key: HADOOP-14441
> URL: https://issues.apache.org/jira/browse/HADOOP-14441
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 2.7.0
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Attachments: HADOOP-14441.001.patch, HADOOP-14441.002.patch
>
>
> LoadBalancingKMSClientProvider only gets delegation token from one KMS 
> instance, in a round-robin fashion. This is arguably a bug, as JavaDoc for 
> {{KeyProviderDelegationTokenExtension#addDelegationTokens}} states:
> {quote}
> /**
>  * The implementer of this class will take a renewer and add all
>  * delegation tokens associated with the renewer to the 
>  * Credentials object if it is not already present, 
> ...
> **/
> {quote}
> This bug doesn't pop up very often, because HDFS clients such as MapReduce 
> unintentionally calls {{FileSystem#addDelegationTokens}} multiple times.
> We have a custom client that accesses HDFS/KMS-HA using delegation token, and 
> we were puzzled why it always throws "Failed to find any Kerberos tgt" 
> exceptions talking to one KMS but not the other. Turns out that client 
> couldn't talk to the KMS because {{FileSystem#addDelegationTokens}} only gets 
> one KMS delegation token at a time.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-14441) LoadBalancingKMSClientProvider#addDelegationTokens should add delegation tokens from all KMS instances

[Yongjun Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-14441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16018026#comment-16018026
 ] 

Yongjun Zhang commented on HADOOP-14441:


The LoadBalancingKMSClientProvider#doOp() only try one index at a time. I don't 
see retry there. Thanks.



> LoadBalancingKMSClientProvider#addDelegationTokens should add delegation 
> tokens from all KMS instances
> --
>
> Key: HADOOP-14441
> URL: https://issues.apache.org/jira/browse/HADOOP-14441
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 2.7.0
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Attachments: HADOOP-14441.001.patch, HADOOP-14441.002.patch
>
>
> LoadBalancingKMSClientProvider only gets delegation token from one KMS 
> instance, in a round-robin fashion. This is arguably a bug, as JavaDoc for 
> {{KeyProviderDelegationTokenExtension#addDelegationTokens}} states:
> {quote}
> /**
>  * The implementer of this class will take a renewer and add all
>  * delegation tokens associated with the renewer to the 
>  * Credentials object if it is not already present, 
> ...
> **/
> {quote}
> This bug doesn't pop up very often, because HDFS clients such as MapReduce 
> unintentionally calls {{FileSystem#addDelegationTokens}} multiple times.
> We have a custom client that accesses HDFS/KMS-HA using delegation token, and 
> we were puzzled why it always throws "Failed to find any Kerberos tgt" 
> exceptions talking to one KMS but not the other. Turns out that client 
> couldn't talk to the KMS because {{FileSystem#addDelegationTokens}} only gets 
> one KMS delegation token at a time.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-14441) LoadBalancingKMSClientProvider#addDelegationTokens should add delegation tokens from all KMS instances

[Wei-Chiu Chuang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-14441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16017995#comment-16017995
 ] 

Wei-Chiu Chuang commented on HADOOP-14441:
--

The LoadBalancingKMSClientProvider#doOp() does so. 

Thanks

> LoadBalancingKMSClientProvider#addDelegationTokens should add delegation 
> tokens from all KMS instances
> --
>
> Key: HADOOP-14441
> URL: https://issues.apache.org/jira/browse/HADOOP-14441
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 2.7.0
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Attachments: HADOOP-14441.001.patch, HADOOP-14441.002.patch
>
>
> LoadBalancingKMSClientProvider only gets delegation token from one KMS 
> instance, in a round-robin fashion. This is arguably a bug, as JavaDoc for 
> {{KeyProviderDelegationTokenExtension#addDelegationTokens}} states:
> {quote}
> /**
>  * The implementer of this class will take a renewer and add all
>  * delegation tokens associated with the renewer to the 
>  * Credentials object if it is not already present, 
> ...
> **/
> {quote}
> This bug doesn't pop up very often, because HDFS clients such as MapReduce 
> unintentionally calls {{FileSystem#addDelegationTokens}} multiple times.
> We have a custom client that accesses HDFS/KMS-HA using delegation token, and 
> we were puzzled why it always throws "Failed to find any Kerberos tgt" 
> exceptions talking to one KMS but not the other. Turns out that client 
> couldn't talk to the KMS because {{FileSystem#addDelegationTokens}} only gets 
> one KMS delegation token at a time.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-14441) LoadBalancingKMSClientProvider#addDelegationTokens should add delegation tokens from all KMS instances

[Yongjun Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-14441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16017994#comment-16017994
 ] 

Yongjun Zhang commented on HADOOP-14441:


But I don't see the catch/retry in Hadoop code. Do we expect client code to do 
so? It seems we should do it within Hadoop. [~jojochuang].
 

> LoadBalancingKMSClientProvider#addDelegationTokens should add delegation 
> tokens from all KMS instances
> --
>
> Key: HADOOP-14441
> URL: https://issues.apache.org/jira/browse/HADOOP-14441
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 2.7.0
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Attachments: HADOOP-14441.001.patch, HADOOP-14441.002.patch
>
>
> LoadBalancingKMSClientProvider only gets delegation token from one KMS 
> instance, in a round-robin fashion. This is arguably a bug, as JavaDoc for 
> {{KeyProviderDelegationTokenExtension#addDelegationTokens}} states:
> {quote}
> /**
>  * The implementer of this class will take a renewer and add all
>  * delegation tokens associated with the renewer to the 
>  * Credentials object if it is not already present, 
> ...
> **/
> {quote}
> This bug doesn't pop up very often, because HDFS clients such as MapReduce 
> unintentionally calls {{FileSystem#addDelegationTokens}} multiple times.
> We have a custom client that accesses HDFS/KMS-HA using delegation token, and 
> we were puzzled why it always throws "Failed to find any Kerberos tgt" 
> exceptions talking to one KMS but not the other. Turns out that client 
> couldn't talk to the KMS because {{FileSystem#addDelegationTokens}} only gets 
> one KMS delegation token at a time.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-14441) LoadBalancingKMSClientProvider#addDelegationTokens should add delegation tokens from all KMS instances

[Wei-Chiu Chuang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-14441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16017975#comment-16017975
 ] 

Wei-Chiu Chuang commented on HADOOP-14441:
--

Hi [~yzhangal] thanks for chiming in.
I've thought about that too, but if renew/cancel fails for a specific KMS, the 
client re-tries with the next KMS in a round robin fashion. So these operations 
will succeed eventually. Renew in particular is issued once in a few days.

> LoadBalancingKMSClientProvider#addDelegationTokens should add delegation 
> tokens from all KMS instances
> --
>
> Key: HADOOP-14441
> URL: https://issues.apache.org/jira/browse/HADOOP-14441
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 2.7.0
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Attachments: HADOOP-14441.001.patch, HADOOP-14441.002.patch
>
>
> LoadBalancingKMSClientProvider only gets delegation token from one KMS 
> instance, in a round-robin fashion. This is arguably a bug, as JavaDoc for 
> {{KeyProviderDelegationTokenExtension#addDelegationTokens}} states:
> {quote}
> /**
>  * The implementer of this class will take a renewer and add all
>  * delegation tokens associated with the renewer to the 
>  * Credentials object if it is not already present, 
> ...
> **/
> {quote}
> This bug doesn't pop up very often, because HDFS clients such as MapReduce 
> unintentionally calls {{FileSystem#addDelegationTokens}} multiple times.
> We have a custom client that accesses HDFS/KMS-HA using delegation token, and 
> we were puzzled why it always throws "Failed to find any Kerberos tgt" 
> exceptions talking to one KMS but not the other. Turns out that client 
> couldn't talk to the KMS because {{FileSystem#addDelegationTokens}} only gets 
> one KMS delegation token at a time.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-14441) LoadBalancingKMSClientProvider#addDelegationTokens should add delegation tokens from all KMS instances

[Yongjun Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-14441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16017966#comment-16017966
 ] 

Yongjun Zhang commented on HADOOP-14441:


No problem [~jojochuang]. 

Thanks for the updated patch. I looked at have a high level comment:

Looks to me that the following operations need to have similar fix, given a 
token to renew and cancel, we can either derive the KMS from the service field 
in the token, and operate on this KMS directly, or use a loop like the one you 
changed with addDelegationToken.

{code}
 @Override
  public long renewDelegationToken(final Token token) throws IOException {
return doOp(new ProviderCallable() {
  @Override
  public Long call(KMSClientProvider provider) throws IOException {
return provider.renewDelegationToken(token);
  }
}, nextIdx());
  }

  @Override
  public Void cancelDelegationToken(final Token token) throws IOException {
return doOp(new ProviderCallable() {
  @Override
  public Void call(KMSClientProvider provider) throws IOException {
provider.cancelDelegationToken(token);
return null;
  }
}, nextIdx());
  }
{code}
Do you agree?

Thanks.


> LoadBalancingKMSClientProvider#addDelegationTokens should add delegation 
> tokens from all KMS instances
> --
>
> Key: HADOOP-14441
> URL: https://issues.apache.org/jira/browse/HADOOP-14441
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 2.7.0
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Attachments: HADOOP-14441.001.patch, HADOOP-14441.002.patch
>
>
> LoadBalancingKMSClientProvider only gets delegation token from one KMS 
> instance, in a round-robin fashion. This is arguably a bug, as JavaDoc for 
> {{KeyProviderDelegationTokenExtension#addDelegationTokens}} states:
> {quote}
> /**
>  * The implementer of this class will take a renewer and add all
>  * delegation tokens associated with the renewer to the 
>  * Credentials object if it is not already present, 
> ...
> **/
> {quote}
> This bug doesn't pop up very often, because HDFS clients such as MapReduce 
> unintentionally calls {{FileSystem#addDelegationTokens}} multiple times.
> We have a custom client that accesses HDFS/KMS-HA using delegation token, and 
> we were puzzled why it always throws "Failed to find any Kerberos tgt" 
> exceptions talking to one KMS but not the other. Turns out that client 
> couldn't talk to the KMS because {{FileSystem#addDelegationTokens}} only gets 
> one KMS delegation token at a time.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-14441) LoadBalancingKMSClientProvider#addDelegationTokens should add delegation tokens from all KMS instances

[Yongjun Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-14441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16017891#comment-16017891
 ] 

Yongjun Zhang commented on HADOOP-14441:


Hi [~jojochuang], Thanks for working on this. Seems you forgot to include the 
fix code in the patch.


> LoadBalancingKMSClientProvider#addDelegationTokens should add delegation 
> tokens from all KMS instances
> --
>
> Key: HADOOP-14441
> URL: https://issues.apache.org/jira/browse/HADOOP-14441
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 2.7.0
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Attachments: HADOOP-14441.001.patch
>
>
> LoadBalancingKMSClientProvider only gets delegation token from one KMS 
> instance, in a round-robin fashion. This is arguably a bug, as JavaDoc for 
> {{KeyProviderDelegationTokenExtension#addDelegationTokens}} states:
> {quote}
> /**
>  * The implementer of this class will take a renewer and add all
>  * delegation tokens associated with the renewer to the 
>  * Credentials object if it is not already present, 
> ...
> **/
> {quote}
> This bug doesn't pop up very often, because HDFS clients such as MapReduce 
> unintentionally calls {{FileSystem#addDelegationTokens}} multiple times.
> We have a custom client that accesses HDFS/KMS-HA using delegation token, and 
> we were puzzled why it always throws "Failed to find any Kerberos tgt" 
> exceptions talking to one KMS but not the other. Turns out that client 
> couldn't talk to the KMS because {{FileSystem#addDelegationTokens}} only gets 
> one KMS delegation token at a time.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org