[jira] [Commented] (HDFS-12600) CLONE - KMSClientProvider won't work with KMS delegation token retrieved from non-Java client.

2017-10-05 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-12600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16194019#comment-16194019
 ] 

Hadoop QA commented on HDFS-12600:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
19s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m  
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  3m  
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
14m 24s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m  
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m  
4s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  2m 
 7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m 
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 13m 
31s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
2m 23s{color} | {color:orange} root: The patch generated 2 new + 772 unchanged 
- 7 fixed = 774 total (was 779) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}  
8m 59s{color} | {color:green} patch has no errors when building and testing our 
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m  
8s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  8m  9s{color} 
| {color:red} hadoop-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 97m 41s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  4m 
20s{color} | {color:green} hadoop-hdfs-httpfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
31s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}202m 39s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.net.TestDNS |
|   | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
|   | hadoop.hdfs.TestReadStripedFileWithMissingBlocks |
|   | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HDFS-12600 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12890600/HDFS-12600.002.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  |
| uname | Linux 2b0aaa19cca0 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 
12:48:20

[jira] [Commented] (HDFS-12600) CLONE - KMSClientProvider won't work with KMS delegation token retrieved from non-Java client.

2017-10-05 Thread Xiaoyu Yao (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-12600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16193812#comment-16193812
 ] 

Xiaoyu Yao commented on HDFS-12600:
---

Agree, thanks [~xiaochen] for the new patch to trigger the hadoop-hdfs test.

> CLONE - KMSClientProvider won't work with KMS delegation token retrieved from 
> non-Java client.
> --
>
> Key: HDFS-12600
> URL: https://issues.apache.org/jira/browse/HDFS-12600
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
> Attachments: HDFS-12600.001.patch, HDFS-12600.002.patch
>
>
> HADOOP-13381 added support to use KMS delegation token to connect to KMS 
> server for key operations. However, the logic to check if the UGI container 
> KMS delegation token assumes that the token must contain a service attribute. 
> Otherwise, a KMS delegation token won't be recognized.
> For delegation token obtained via non-java client such curl (http), the 
> default DelegationTokenAuthenticationHandler only support *renewer* parameter 
> and assume the client itself will add the service attribute. This makes a 
> java client with KMSClientProvdier can't use for KMS delegation token 
> retrieved form non-java client because the token does not contain a service 
> attribute. 
> I did some investigation on this and found two solutions:
> 1. Similar use case exists for webhdfs, and webhdfs supports it with a 
> ["service" 
> parameter|https://hadoop.apache.org/docs/r2.8.0/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Get_Delegation_Token].
> We can do this similarly by allowing client to specify a service attribute in 
> the request URL and included in the token returned like webhdfs. Even though 
> this will change in DelegationTokenAuthenticationHandler and may affect many 
> other web component,  this seems to be a clean and low risk solution because 
> it will be an optional parameter. Also, other components get non-java client 
> interop support for free if they have the similar use case. 
> 2. The other way to solve this is to release the token check in 
> KMSClientProvider to check only the token kind instead of the service.  This 
> is an easy work around but seems less optimal to me. 
> cc: [~xiaochen] for additional input.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-12600) CLONE - KMSClientProvider won't work with KMS delegation token retrieved from non-Java client.

2017-10-05 Thread Xiao Chen (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-12600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16193808#comment-16193808
 ] 

Xiao Chen commented on HDFS-12600:
--

+1  unit7m 54s  hadoop-common in the patch passed.

Hi [~xyao], thanks for the jira. I think we still need the hdfs dummy change to 
let jenkins actually run that component. Attaching a patch for that...

> CLONE - KMSClientProvider won't work with KMS delegation token retrieved from 
> non-Java client.
> --
>
> Key: HDFS-12600
> URL: https://issues.apache.org/jira/browse/HDFS-12600
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
> Attachments: HDFS-12600.001.patch
>
>
> HADOOP-13381 added support to use KMS delegation token to connect to KMS 
> server for key operations. However, the logic to check if the UGI container 
> KMS delegation token assumes that the token must contain a service attribute. 
> Otherwise, a KMS delegation token won't be recognized.
> For delegation token obtained via non-java client such curl (http), the 
> default DelegationTokenAuthenticationHandler only support *renewer* parameter 
> and assume the client itself will add the service attribute. This makes a 
> java client with KMSClientProvdier can't use for KMS delegation token 
> retrieved form non-java client because the token does not contain a service 
> attribute. 
> I did some investigation on this and found two solutions:
> 1. Similar use case exists for webhdfs, and webhdfs supports it with a 
> ["service" 
> parameter|https://hadoop.apache.org/docs/r2.8.0/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Get_Delegation_Token].
> We can do this similarly by allowing client to specify a service attribute in 
> the request URL and included in the token returned like webhdfs. Even though 
> this will change in DelegationTokenAuthenticationHandler and may affect many 
> other web component,  this seems to be a clean and low risk solution because 
> it will be an optional parameter. Also, other components get non-java client 
> interop support for free if they have the similar use case. 
> 2. The other way to solve this is to release the token check in 
> KMSClientProvider to check only the token kind instead of the service.  This 
> is an easy work around but seems less optimal to me. 
> cc: [~xiaochen] for additional input.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-12600) CLONE - KMSClientProvider won't work with KMS delegation token retrieved from non-Java client.

2017-10-05 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-12600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16193792#comment-16193792
 ] 

Hadoop QA commented on HDFS-12600:
--

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m 
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m 
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
10m 22s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
56s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m  
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m  
7s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 38s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch 
generated 2 new + 69 unchanged - 7 fixed = 71 total (was 76) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}  
8m 56s{color} | {color:green} patch has no errors when building and testing our 
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  7m 
54s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
31s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 75m 49s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HDFS-12600 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12890579/HDFS-12600.001.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  |
| uname | Linux dfa6bfd6d579 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / d0046bb |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-HDFS-Build/21539/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/21539/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U: 
hadoop-common-project/hadoop-common |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/21539/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> CLONE - KMSClientProvider won't