subject:"\[jira\] \[Commented\] \(HADOOP\-15954\) ABFS\: Enable owner and group conversion for MSI and login user using OAuth"

[jira] [Commented] (HADOOP-15954) ABFS: Enable owner and group conversion for MSI and login user using OAuth

2019-03-05 Thread Steve Loughran (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784308#comment-16784308
 ] 

Steve Loughran commented on HADOOP-15954:
-

backport to 3.2; ran new test first

> ABFS: Enable owner and group conversion for MSI and login user using OAuth
> --
>
> Key: HADOOP-15954
> URL: https://issues.apache.org/jira/browse/HADOOP-15954
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: junhua gu
>Assignee: Da Zhou
>Priority: Major
> Fix For: 3.3.0, 3.2.1
>
> Attachments: HADOOP-15954-001.patch, HADOOP-15954-002.patch, 
> HADOOP-15954-003.patch, HADOOP-15954-004.patch, HADOOP-15954-005.patch, 
> HADOOP-15954-006.patch, HADOOP-15954-007.patch, HADOOP-15954-008.patch, 
> HADOOP-15954-009.patch, HADOOP-15954-010.patch, HADOOP-15954-011.patch, 
> HADOOP-15954-012.patch, HADOOP-15954-013.patch
>
>
> Add support for overwriting owner and group in set/get operations to be the 
> service principal id when OAuth is used. Add support for upn short name 
> format.
>  
> Add Standard Transformer for SharedKey / Service 
> Add interface provides an extensible model for customizing the acquisition of 
> Identity Transformer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15954) ABFS: Enable owner and group conversion for MSI and login user using OAuth

2019-02-07 Thread Hudson (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16763128#comment-16763128
 ] 

Hudson commented on HADOOP-15954:
-

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15911 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/15911/])
Revert "HADOOP-15954. ABFS: Enable owner and group conversion for MSI (stevel: 
rev 668817a6cefa6025ddfe082ed71d7d317d811381)
* (delete) 
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/oauth2/IdentityTransformer.java
* (edit) 
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testErasureCodingConf.xml
* (edit) 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/shell/TestLs.java
* (edit) 
hadoop-common-project/hadoop-common/src/site/markdown/FileSystemShell.md
* (edit) hadoop-common-project/hadoop-common/src/test/resources/testConf.xml
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestStoragePolicyCommands.java
* (delete) 
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsIdentityTransformer.java
* (edit) 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Ls.java
* (edit) 
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/ConfigurationKeys.java
* (edit) 
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/oauth2/package-info.java
* (edit) 
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java
* (edit) hadoop-tools/hadoop-azure/src/site/markdown/testing_azure.md
* (edit) 
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java
HADOOP-15954. ABFS: Enable owner and group conversion for MSI and login 
(stevel: rev 1f1655028eede24197705a594b6ef19e6737db35)
* (edit) 
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java
* (edit) 
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java
* (edit) 
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java
* (edit) hadoop-tools/hadoop-azure/src/site/markdown/testing_azure.md
* (add) 
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsIdentityTransformer.java
* (edit) 
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/ConfigurationKeys.java
* (add) 
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/oauth2/IdentityTransformer.java


> ABFS: Enable owner and group conversion for MSI and login user using OAuth
> --
>
> Key: HADOOP-15954
> URL: https://issues.apache.org/jira/browse/HADOOP-15954
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: junhua gu
>Assignee: Da Zhou
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-15954-001.patch, HADOOP-15954-002.patch, 
> HADOOP-15954-003.patch, HADOOP-15954-004.patch, HADOOP-15954-005.patch, 
> HADOOP-15954-006.patch, HADOOP-15954-007.patch, HADOOP-15954-008.patch, 
> HADOOP-15954-009.patch, HADOOP-15954-010.patch, HADOOP-15954-011.patch, 
> HADOOP-15954-012.patch, HADOOP-15954-013.patch
>
>
> Add support for overwriting owner and group in set/get operations to be the 
> service principal id when OAuth is used. Add support for upn short name 
> format.
>  
> Add Standard Transformer for SharedKey / Service 
> Add interface provides an extensible model for customizing the acquisition of 
> Identity Transformer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15954) ABFS: Enable owner and group conversion for MSI and login user using OAuth

2019-02-07 Thread Steve Loughran (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16763120#comment-16763120
 ] 

Steve Loughran commented on HADOOP-15954:
-

ok, two new patches : one to revert, one for minimal apply.

(I'm getting patches wrong today. I'm going to take a break from committing 
patches for the rest of the week to make sure I don't make anything worse)

> ABFS: Enable owner and group conversion for MSI and login user using OAuth
> --
>
> Key: HADOOP-15954
> URL: https://issues.apache.org/jira/browse/HADOOP-15954
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: junhua gu
>Assignee: Da Zhou
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-15954-001.patch, HADOOP-15954-002.patch, 
> HADOOP-15954-003.patch, HADOOP-15954-004.patch, HADOOP-15954-005.patch, 
> HADOOP-15954-006.patch, HADOOP-15954-007.patch, HADOOP-15954-008.patch, 
> HADOOP-15954-009.patch, HADOOP-15954-010.patch, HADOOP-15954-011.patch, 
> HADOOP-15954-012.patch, HADOOP-15954-013.patch
>
>
> Add support for overwriting owner and group in set/get operations to be the 
> service principal id when OAuth is used. Add support for upn short name 
> format.
>  
> Add Standard Transformer for SharedKey / Service 
> Add interface provides an extensible model for customizing the acquisition of 
> Identity Transformer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15954) ABFS: Enable owner and group conversion for MSI and login user using OAuth

2019-02-07 Thread Steve Loughran (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16763116#comment-16763116
 ] 

Steve Loughran commented on HADOOP-15954:
-

oh no, mixed two patches together. Will revert and reapply this only. sorry

> ABFS: Enable owner and group conversion for MSI and login user using OAuth
> --
>
> Key: HADOOP-15954
> URL: https://issues.apache.org/jira/browse/HADOOP-15954
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: junhua gu
>Assignee: Da Zhou
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-15954-001.patch, HADOOP-15954-002.patch, 
> HADOOP-15954-003.patch, HADOOP-15954-004.patch, HADOOP-15954-005.patch, 
> HADOOP-15954-006.patch, HADOOP-15954-007.patch, HADOOP-15954-008.patch, 
> HADOOP-15954-009.patch, HADOOP-15954-010.patch, HADOOP-15954-011.patch, 
> HADOOP-15954-012.patch, HADOOP-15954-013.patch
>
>
> Add support for overwriting owner and group in set/get operations to be the 
> service principal id when OAuth is used. Add support for upn short name 
> format.
>  
> Add Standard Transformer for SharedKey / Service 
> Add interface provides an extensible model for customizing the acquisition of 
> Identity Transformer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15954) ABFS: Enable owner and group conversion for MSI and login user using OAuth

2019-02-06 Thread Surendra Singh Lilhore (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16762381#comment-16762381
 ] 

Surendra Singh Lilhore commented on HADOOP-15954:
-

Hi [~ste...@apache.org],

I think you merged HADOOP-16077 with this jira. Pls check 
[commit|https://github.com/apache/hadoop/commit/fa8cd1bf28f5b81849ba351a2d7225fbc580350d#diff-a9b959cf40730e23c0438952fb843d2f].

> ABFS: Enable owner and group conversion for MSI and login user using OAuth
> --
>
> Key: HADOOP-15954
> URL: https://issues.apache.org/jira/browse/HADOOP-15954
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: junhua gu
>Assignee: Da Zhou
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-15954-001.patch, HADOOP-15954-002.patch, 
> HADOOP-15954-003.patch, HADOOP-15954-004.patch, HADOOP-15954-005.patch, 
> HADOOP-15954-006.patch, HADOOP-15954-007.patch, HADOOP-15954-008.patch, 
> HADOOP-15954-009.patch, HADOOP-15954-010.patch, HADOOP-15954-011.patch, 
> HADOOP-15954-012.patch, HADOOP-15954-013.patch
>
>
> Add support for overwriting owner and group in set/get operations to be the 
> service principal id when OAuth is used. Add support for upn short name 
> format.
>  
> Add Standard Transformer for SharedKey / Service 
> Add interface provides an extensible model for customizing the acquisition of 
> Identity Transformer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15954) ABFS: Enable owner and group conversion for MSI and login user using OAuth

2019-02-05 Thread Hudson (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16761146#comment-16761146
 ] 

Hudson commented on HADOOP-15954:
-

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15894 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/15894/])
HADOOP-15954. ABFS: Enable owner and group conversion for MSI and login 
(stevel: rev fa8cd1bf28f5b81849ba351a2d7225fbc580350d)
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestStoragePolicyCommands.java
* (edit) 
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java
* (edit) 
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java
* (add) 
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/oauth2/IdentityTransformer.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testErasureCodingConf.xml
* (edit) hadoop-common-project/hadoop-common/src/test/resources/testConf.xml
* (edit) 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/shell/TestLs.java
* (edit) 
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java
* (edit) 
hadoop-common-project/hadoop-common/src/site/markdown/FileSystemShell.md
* (edit) 
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/oauth2/package-info.java
* (add) 
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsIdentityTransformer.java
* (edit) 
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/ConfigurationKeys.java
* (edit) 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Ls.java
* (edit) hadoop-tools/hadoop-azure/src/site/markdown/testing_azure.md


> ABFS: Enable owner and group conversion for MSI and login user using OAuth
> --
>
> Key: HADOOP-15954
> URL: https://issues.apache.org/jira/browse/HADOOP-15954
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: junhua gu
>Assignee: Da Zhou
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-15954-001.patch, HADOOP-15954-002.patch, 
> HADOOP-15954-003.patch, HADOOP-15954-004.patch, HADOOP-15954-005.patch, 
> HADOOP-15954-006.patch, HADOOP-15954-007.patch, HADOOP-15954-008.patch, 
> HADOOP-15954-009.patch, HADOOP-15954-010.patch, HADOOP-15954-011.patch, 
> HADOOP-15954-012.patch, HADOOP-15954-013.patch
>
>
> Add support for overwriting owner and group in set/get operations to be the 
> service principal id when OAuth is used. Add support for upn short name 
> format.
>  
> Add Standard Transformer for SharedKey / Service 
> Add interface provides an extensible model for customizing the acquisition of 
> Identity Transformer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15954) ABFS: Enable owner and group conversion for MSI and login user using OAuth

2019-02-05 Thread Steve Loughran (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16761136#comment-16761136
 ] 

Steve Loughran commented on HADOOP-15954:
-

BTW, gave both of you credit for the work. thanks!

> ABFS: Enable owner and group conversion for MSI and login user using OAuth
> --
>
> Key: HADOOP-15954
> URL: https://issues.apache.org/jira/browse/HADOOP-15954
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: junhua gu
>Assignee: Da Zhou
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-15954-001.patch, HADOOP-15954-002.patch, 
> HADOOP-15954-003.patch, HADOOP-15954-004.patch, HADOOP-15954-005.patch, 
> HADOOP-15954-006.patch, HADOOP-15954-007.patch, HADOOP-15954-008.patch, 
> HADOOP-15954-009.patch, HADOOP-15954-010.patch, HADOOP-15954-011.patch, 
> HADOOP-15954-012.patch, HADOOP-15954-013.patch
>
>
> Add support for overwriting owner and group in set/get operations to be the 
> service principal id when OAuth is used. Add support for upn short name 
> format.
>  
> Add Standard Transformer for SharedKey / Service 
> Add interface provides an extensible model for customizing the acquisition of 
> Identity Transformer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15954) ABFS: Enable owner and group conversion for MSI and login user using OAuth

2019-02-01 Thread Hadoop QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16758697#comment-16758697
 ] 

Hadoop QA commented on HADOOP-15954:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m 
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
13m 45s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
21s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
24s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 16s{color} | {color:orange} hadoop-tools/hadoop-azure: The patch generated 2 
new + 3 unchanged - 0 fixed = 5 total (was 3) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
14m 10s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m 
19s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
27s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 57m 16s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HADOOP-15954 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12957317/HADOOP-15954-013.patch
 |
| Optional Tests |  dupname  asflicense  compile  javac  javadoc  mvninstall  
mvnsite  unit  shadedclient  findbugs  checkstyle  |
| uname | Linux 2b9fa9c51577 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct 
5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 4123353 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/15875/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/15875/testReport/ |
| Max. process+thread count | 306 (vs. ulimit of 1) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/15875/console |
| Powered by | Apache Yetus 0.8.0   http://yetus.apache.org

[jira] [Commented] (HADOOP-15954) ABFS: Enable owner and group conversion for MSI and login user using OAuth

2019-02-01 Thread Da Zhou (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16758661#comment-16758661
 ] 

Da Zhou commented on HADOOP-15954:
--

After reconsideration, we decided not to make the Identity transformer a plugin 
mode at this moment. So I revert it back from an interface to a normal class.
The main reason are: 
*1*. The proposed interface lead to performance issue, it forced the ABFS to 
always do extra allocation (eg: In getFileStatus(), need to always create file 
status and then pass it to transformer, which might create new FileStatus if 
the identity needs to be transformed.)
*2*.Changing the interface to focus on only own/group transformation can help, 
but the interface becomes complex and less flexible for users, which betrays 
the original goal.

Submitting patch 013, changes are:
1. Removed the interface.
1. Move the UsergroupInformation  parsing and exception handling to 
AzureBlobFileSystemStore() to avoid duplicate code and inconsistent local 
owner, group located in AzureBlobFileSystem and IdentityTransformer. 
2. Refactored  "IdentityTransformer" to make it more readable.
3. Unit tests are added for each identity transformation scenario. 



> ABFS: Enable owner and group conversion for MSI and login user using OAuth
> --
>
> Key: HADOOP-15954
> URL: https://issues.apache.org/jira/browse/HADOOP-15954
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: junhua gu
>Assignee: Da Zhou
>Priority: Major
> Attachments: HADOOP-15954-001.patch, HADOOP-15954-002.patch, 
> HADOOP-15954-003.patch, HADOOP-15954-004.patch, HADOOP-15954-005.patch, 
> HADOOP-15954-006.patch, HADOOP-15954-007.patch, HADOOP-15954-008.patch, 
> HADOOP-15954-009.patch, HADOOP-15954-010.patch, HADOOP-15954-011.patch, 
> HADOOP-15954-012.patch
>
>
> Add support for overwriting owner and group in set/get operations to be the 
> service principal id when OAuth is used. Add support for upn short name 
> format.
>  
> Add Standard Transformer for SharedKey / Service 
> Add interface provides an extensible model for customizing the acquisition of 
> Identity Transformer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15954) ABFS: Enable owner and group conversion for MSI and login user using OAuth

2019-01-28 Thread Da Zhou (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16754212#comment-16754212
 ] 

Da Zhou commented on HADOOP-15954:
--

Thanks for pointing out!  Yes I realized some issues mentioned above so I was 
working on the improvement.
 [~ste...@apache.org]
 *CustomIdentityTransformer*: Sure, I will add filesystem URI and close().
 *transformListStatus(FileStatus[])*: I plan to remove this from interface 
because it causes extra allocation, and this could be avoided by just declaring 
sth like "*void transformUserAndGroupForWriteOperation(String[]userGroup))*" 
and "*void transformUserAndGroupForReadOperation(String[]userGroup))*", so the 
updated owner and group can be used to create FileStatus/AclStatus, as 
mentioned by [~tmarquardt] for L117.
Will update the logic expression to improve readability.
 *ITestAbfsIdentityTransformer*: will do the refactoring.

[~tmarquardt] 
DefaultSPIdentityTransformer.java:
L84-92: Yes the use/group is inconsistent when exception happens when parsing 
userGroupInfo in AzureBlobFileSystem and DefaultIdentityTransformer.
L107-114: once the owner meet the first if condition and get transformed to 
localUser(which is already a shortUserName), then checking  
shouldUseShortUserName() will be redundant, that is the reason it is put in the 
else. 
L117, L125-129, L157: these will be resolved. "void 
transformListStatus(FileStatus[] entries)" makes sense.

> ABFS: Enable owner and group conversion for MSI and login user using OAuth
> --
>
> Key: HADOOP-15954
> URL: https://issues.apache.org/jira/browse/HADOOP-15954
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: junhua gu
>Assignee: Da Zhou
>Priority: Major
> Attachments: HADOOP-15954-001.patch, HADOOP-15954-002.patch, 
> HADOOP-15954-003.patch, HADOOP-15954-004.patch, HADOOP-15954-005.patch, 
> HADOOP-15954-006.patch, HADOOP-15954-007.patch, HADOOP-15954-008.patch, 
> HADOOP-15954-009.patch, HADOOP-15954-010.patch, HADOOP-15954-011.patch, 
> HADOOP-15954-012.patch
>
>
> Add support for overwriting owner and group in set/get operations to be the 
> service principal id when OAuth is used. Add support for upn short name 
> format.
>  
> Add Standard Transformer for SharedKey / Service 
> Add interface provides an extensible model for customizing the acquisition of 
> Identity Transformer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15954) ABFS: Enable owner and group conversion for MSI and login user using OAuth

2019-01-28 Thread Steve Loughran (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16754125#comment-16754125
 ] 

Steve Loughran commented on HADOOP-15954:
-

One other thing to consider, ABFS doesn't *yet* implement listFiles(path, 
recursive), so the flat listing is implemented client side as a treewalk.

If ABFS were ever to implement that through ABFS store requests which are more 
efficient at enumerating all objects under a path, then this same translation 
will be needed. Again, this argues for doing the conversion early enough that 
however a file status is retrieved, it gets the mapped values

> ABFS: Enable owner and group conversion for MSI and login user using OAuth
> --
>
> Key: HADOOP-15954
> URL: https://issues.apache.org/jira/browse/HADOOP-15954
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: junhua gu
>Assignee: Da Zhou
>Priority: Major
> Attachments: HADOOP-15954-001.patch, HADOOP-15954-002.patch, 
> HADOOP-15954-003.patch, HADOOP-15954-004.patch, HADOOP-15954-005.patch, 
> HADOOP-15954-006.patch, HADOOP-15954-007.patch, HADOOP-15954-008.patch, 
> HADOOP-15954-009.patch, HADOOP-15954-010.patch, HADOOP-15954-011.patch, 
> HADOOP-15954-012.patch
>
>
> Add support for overwriting owner and group in set/get operations to be the 
> service principal id when OAuth is used. Add support for upn short name 
> format.
>  
> Add Standard Transformer for SharedKey / Service 
> Add interface provides an extensible model for customizing the acquisition of 
> Identity Transformer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15954) ABFS: Enable owner and group conversion for MSI and login user using OAuth

2019-01-27 Thread Thomas Marquardt (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16753623#comment-16753623
 ] 

Thomas Marquardt commented on HADOOP-15954:
---

*CustomIdentityTransformer.java*:

L94 - replace groupName with userName in the comment

*DefaultSPIdentityTransformer.java*:

L49: We can remove SP from the name since i) people will not know this is an 
abbreviation for service principal and ii) this class does more. 
 I recommend naming it DefaultIdentityTransformer.

L84-92: There is code in AzureBlobFileSystem.initiailize that sets user and 
principalUserGroup, and also changes that behavior based on config settings. 
 It seems we should pass the values for user and group into 
DefaultIdentityTransformer.initialize.

L107-114: Should the short name conversion be done outside the else?

L117: We should avoid the extra FileStatus allocation, especially since this is 
used for list operations. These allocations can add up.
 Since the interface is really about the ability to transform identities, I am 
wondering why we allow the interface to replace the entire FileStatus? 
 Looking at AzureBlobFileSystemStore.getFileStatus and listFileStatus, it 
appears we could replace the owner and owning group before we create the file
 status, which would avoid extra allocations, especially for list.

L125-129: If for some reason this function is not removed based on the comment 
above...note that replacement can be done in place using the current array, and 
 thereby avoid a potentially large allocation. Just change the signature to 
void transformListStatus(FileStatus[] entries).

L157: Similar to comment L117 above, why not convert the owner and owning group 
inside AzureBlobFileSystemStore.getAclStatus before the AclStatus is created?  
This would avoid the extra AclStatus allocation and builder.

> ABFS: Enable owner and group conversion for MSI and login user using OAuth
> --
>
> Key: HADOOP-15954
> URL: https://issues.apache.org/jira/browse/HADOOP-15954
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: junhua gu
>Assignee: Da Zhou
>Priority: Major
> Attachments: HADOOP-15954-001.patch, HADOOP-15954-002.patch, 
> HADOOP-15954-003.patch, HADOOP-15954-004.patch, HADOOP-15954-005.patch, 
> HADOOP-15954-006.patch, HADOOP-15954-007.patch, HADOOP-15954-008.patch, 
> HADOOP-15954-009.patch, HADOOP-15954-010.patch, HADOOP-15954-011.patch, 
> HADOOP-15954-012.patch
>
>
> Add support for overwriting owner and group in set/get operations to be the 
> service principal id when OAuth is used. Add support for upn short name 
> format.
>  
> Add Standard Transformer for SharedKey / Service 
> Add interface provides an extensible model for customizing the acquisition of 
> Identity Transformer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15954) ABFS: Enable owner and group conversion for MSI and login user using OAuth

2019-01-27 Thread Steve Loughran (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16753400#comment-16753400
 ] 

Steve Loughran commented on HADOOP-15954:
-



h3. CustomIdentityTransformer

* initialize() method should take a filesystem URI. This lines it up with what 
I'm doing in HADOOP-16088, and allows for auth/delegation token/identity 
providers to do different things based on the URI of the filesystem.

* have it extend Closeable and have the base implementation have an empty 
close() call. Allows for transformers to create state (network connections, 
threads etc) and clean up in the close. FS can then invoke the close() call in 
its own close() operation, which will be invoked in a shared service when 
FileSystem.closeAllForUser(UGI) is invoked.

Should {{transformListStatus(FileStatus[])} take an return lists? Or is it 
mutating the array passed in? If that is allowed, then it should be mentioned, 
and if not, then not. I'm just thinking about what is a good combination of 
efficiency and integration.

h3. DefaultSPIdentityTransformer.transformFileStatus

L103: Can put braces around the second || clause, so the precedence of 
operators is obvious (ie. that the || clause includes the && comparison). I 
know it's something "obvious" we are all meant to know, but I did actually have 
to look it up myself to makes sure. 

{code}
if (shouldReplaceOwnerAndGroup(replaceOwner) || SUPER_USER.equals(replaceOwner) 
&& !skipSuperUserReplacement) {
{code}

to 

{code}
if (shouldReplaceOwnerAndGroup(replaceOwner)
   || (SUPER_USER.equals(replaceOwner) && !skipSuperUserReplacement)) {
{code}


h3. ITestAbfsIdentityTransformer

* Can you refactor those assertEquals calls in the first test methods, 
testDaemonServiceSettingIdentity and testFullyQualifiedNameWhenSettingIdentity 
to an equivalent of {{assertIdentityEquals}}

* The code in DefaultSPIdentityTransformer.getShortName(String userName) to 
handle rule transformation seems complex enough to merit testing, somehow.

> ABFS: Enable owner and group conversion for MSI and login user using OAuth
> --
>
> Key: HADOOP-15954
> URL: https://issues.apache.org/jira/browse/HADOOP-15954
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: junhua gu
>Assignee: Da Zhou
>Priority: Major
> Attachments: HADOOP-15954-001.patch, HADOOP-15954-002.patch, 
> HADOOP-15954-003.patch, HADOOP-15954-004.patch, HADOOP-15954-005.patch, 
> HADOOP-15954-006.patch, HADOOP-15954-007.patch, HADOOP-15954-008.patch, 
> HADOOP-15954-009.patch, HADOOP-15954-010.patch, HADOOP-15954-011.patch, 
> HADOOP-15954-012.patch
>
>
> Add support for overwriting owner and group in set/get operations to be the 
> service principal id when OAuth is used. Add support for upn short name 
> format.
>  
> Add Standard Transformer for SharedKey / Service 
> Add interface provides an extensible model for customizing the acquisition of 
> Identity Transformer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15954) ABFS: Enable owner and group conversion for MSI and login user using OAuth

2019-01-27 Thread Steve Loughran (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16753369#comment-16753369
 ] 

Steve Loughran commented on HADOOP-15954:
-

checkstyle complains are minor, but do need fixing
{code}
./hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsIdentityTransformer.java:47:
  private final static String DAEMON_USER = "daemon";:17: 'static' modifier out 
of order with the JLS suggestions. [ModifierOrder]
./hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsIdentityTransformer.java:48:
  private final static String DAEMON_GROUP = "daemon";:17: 'static' modifier 
out of order with the JLS suggestions. [ModifierOrder]
./hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsIdentityTransformer.java:49:
  private final static String ASTERISK = "*";:17: 'static' modifier out of 
order with the JLS suggestions. [ModifierOrder]
{code}

> ABFS: Enable owner and group conversion for MSI and login user using OAuth
> --
>
> Key: HADOOP-15954
> URL: https://issues.apache.org/jira/browse/HADOOP-15954
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: junhua gu
>Assignee: Da Zhou
>Priority: Major
> Attachments: HADOOP-15954-001.patch, HADOOP-15954-002.patch, 
> HADOOP-15954-003.patch, HADOOP-15954-004.patch, HADOOP-15954-005.patch, 
> HADOOP-15954-006.patch, HADOOP-15954-007.patch, HADOOP-15954-008.patch, 
> HADOOP-15954-009.patch, HADOOP-15954-010.patch, HADOOP-15954-011.patch, 
> HADOOP-15954-012.patch
>
>
> Add support for overwriting owner and group in set/get operations to be the 
> service principal id when OAuth is used. Add support for upn short name 
> format.
>  
> Add Standard Transformer for SharedKey / Service 
> Add interface provides an extensible model for customizing the acquisition of 
> Identity Transformer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15954) ABFS: Enable owner and group conversion for MSI and login user using OAuth

2019-01-23 Thread Hadoop QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16750617#comment-16750617
 ] 

Hadoop QA commented on HADOOP-15954:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m 
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
12m 49s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
21s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
23s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 16s{color} | {color:orange} hadoop-tools/hadoop-azure: The patch generated 3 
new + 2 unchanged - 1 fixed = 5 total (was 3) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
13m 59s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m 
17s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
33s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 57m 50s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HADOOP-15954 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12956063/HADOOP-15954-012.patch
 |
| Optional Tests |  dupname  asflicense  compile  javac  javadoc  mvninstall  
mvnsite  unit  shadedclient  findbugs  checkstyle  |
| uname | Linux 1c7c5c1efc35 4.4.0-139-generic #165~14.04.1-Ubuntu SMP Wed Oct 
31 10:55:11 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / f3e642d |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/15833/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/15833/testReport/ |
| Max. process+thread count | 340 (vs. ulimit of 1) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/15833/console |
| Powered by | Apache Yetus 0.8.0   http://yetus.apache.org

[jira] [Commented] (HADOOP-15954) ABFS: Enable owner and group conversion for MSI and login user using OAuth

2019-01-23 Thread Hadoop QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16750605#comment-16750605
 ] 

Hadoop QA commented on HADOOP-15954:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m 
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
11m 42s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
21s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
23s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 15s{color} | {color:orange} hadoop-tools/hadoop-azure: The patch generated 3 
new + 2 unchanged - 1 fixed = 5 total (was 3) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
12m 26s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m 
17s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
23s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 51m 44s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HADOOP-15954 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12956057/HADOOP-15954-012.patch
 |
| Optional Tests |  dupname  asflicense  compile  javac  javadoc  mvninstall  
mvnsite  unit  shadedclient  findbugs  checkstyle  |
| uname | Linux 788b2c8fa510 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / f3e642d |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/15832/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/15832/testReport/ |
| Max. process+thread count | 447 (vs. ulimit of 1) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/15832/console |
| Powered by | Apache Yetus 0.8.0   http://yetus.apache.org |


This

[jira] [Commented] (HADOOP-15954) ABFS: Enable owner and group conversion for MSI and login user using OAuth

2019-01-23 Thread Da Zhou (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16750578#comment-16750578
 ] 

Da Zhou commented on HADOOP-15954:
--

Added test "*ITestAbfsIdentityTransformer*" for *DefaultSPIdentityTransformer*.

All tests passed:
XNS account, using OAuth
Tests run: 35, Failures: 0, Errors: 0, Skipped: 0
Tests run: 333, Failures: 0, Errors: 0, Skipped: 23
Tests run: 168, Failures: 0, Errors: 0, Skipped: 21

XNS account, using SharedKey:
Tests run: 35, Failures: 0, Errors: 0, Skipped: 0
 Tests run: 333, Failures: 0, Errors: 0, Skipped: 21
Tests run: 168, Failures: 0, Errors: 0, Skipped: 15
 
non-XNS account, using Sharedkey:
Tests run: 35, Failures: 0, Errors: 0, Skipped: 0
Tests run: 333, Failures: 0, Errors: 0, Skipped: 207
Tests run: 168, Failures: 0, Errors: 0, Skipped: 15

> ABFS: Enable owner and group conversion for MSI and login user using OAuth
> --
>
> Key: HADOOP-15954
> URL: https://issues.apache.org/jira/browse/HADOOP-15954
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: junhua gu
>Assignee: Da Zhou
>Priority: Major
> Attachments: HADOOP-15954-001.patch, HADOOP-15954-002.patch, 
> HADOOP-15954-003.patch, HADOOP-15954-004.patch, HADOOP-15954-005.patch, 
> HADOOP-15954-006.patch, HADOOP-15954-007.patch, HADOOP-15954-008.patch, 
> HADOOP-15954-009.patch, HADOOP-15954-010.patch, HADOOP-15954-011.patch, 
> HADOOP-15954-012.patch
>
>
> Add support for overwriting owner and group in set/get operations to be the 
> service principal id when OAuth is used. Add support for upn short name 
> format.
>  
> Add Standard Transformer for SharedKey / Service 
> Add interface provides an extensible model for customizing the acquisition of 
> Identity Transformer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15954) ABFS: Enable owner and group conversion for MSI and login user using OAuth

2019-01-22 Thread junhua gu (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748868#comment-16748868
 ] 

junhua gu commented on HADOOP-15954:


*AbfsConfiguration.java*

minor: public String getString(String key, String defaultValue)  can be removed 
since it is no longer used.

The rest looks good to me, +1

> ABFS: Enable owner and group conversion for MSI and login user using OAuth
> --
>
> Key: HADOOP-15954
> URL: https://issues.apache.org/jira/browse/HADOOP-15954
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: junhua gu
>Assignee: Da Zhou
>Priority: Major
> Attachments: HADOOP-15954-001.patch, HADOOP-15954-002.patch, 
> HADOOP-15954-003.patch, HADOOP-15954-004.patch, HADOOP-15954-005.patch, 
> HADOOP-15954-006.patch, HADOOP-15954-007.patch, HADOOP-15954-008.patch, 
> HADOOP-15954-009.patch, HADOOP-15954-010.patch, HADOOP-15954-011.patch, 
> HADOOP-15954-012.patch
>
>
> Add support for overwriting owner and group in set/get operations to be the 
> service principal id when OAuth is used. Add support for upn short name 
> format.
>  
> Add Standard Transformer for SharedKey / Service 
> Add interface provides an extensible model for customizing the acquisition of 
> Identity Transformer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15954) ABFS: Enable owner and group conversion for MSI and login user using OAuth

2019-01-16 Thread Hadoop QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744490#comment-16744490
 ] 

Hadoop QA commented on HADOOP-15954:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m 
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
14m 10s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
24s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
20s{color} | {color:green} hadoop-tools/hadoop-azure: The patch generated 0 new 
+ 2 unchanged - 1 fixed = 2 total (was 3) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
15m 15s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m 
20s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
28s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 58m 40s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HADOOP-15954 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12955145/HADOOP-15954-012.patch
 |
| Optional Tests |  dupname  asflicense  compile  javac  javadoc  mvninstall  
mvnsite  unit  shadedclient  findbugs  checkstyle  |
| uname | Linux 2b45d6bf0086 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct 
5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / f048512 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/15795/testReport/ |
| Max. process+thread count | 316 (vs. ulimit of 1) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/15795/console |
| Powered by | Apache Yetus 0.8.0   http://yetus.apache.org |


This

[jira] [Commented] (HADOOP-15954) ABFS: Enable owner and group conversion for MSI and login user using OAuth

2019-01-16 Thread Da Zhou (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744464#comment-16744464
 ] 

Da Zhou commented on HADOOP-15954:
--

Updates 012:
- Renamed "skipTransFormation" to "skipUserIdentityReplacement ".
- Renamed "enableSuperUserReplacement " to "skipSuperUserReplacement ", updated 
related config keys and default value and the condition check logic.

> ABFS: Enable owner and group conversion for MSI and login user using OAuth
> --
>
> Key: HADOOP-15954
> URL: https://issues.apache.org/jira/browse/HADOOP-15954
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: junhua gu
>Assignee: Da Zhou
>Priority: Major
> Attachments: HADOOP-15954-001.patch, HADOOP-15954-002.patch, 
> HADOOP-15954-003.patch, HADOOP-15954-004.patch, HADOOP-15954-005.patch, 
> HADOOP-15954-006.patch, HADOOP-15954-007.patch, HADOOP-15954-008.patch, 
> HADOOP-15954-009.patch, HADOOP-15954-010.patch, HADOOP-15954-011.patch, 
> HADOOP-15954-012.patch
>
>
> Add support for overwriting owner and group in set/get operations to be the 
> service principal id when OAuth is used. Add support for upn short name 
> format.
>  
> Add Standard Transformer for SharedKey / Service 
> Add interface provides an extensible model for customizing the acquisition of 
> Identity Transformer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15954) ABFS: Enable owner and group conversion for MSI and login user using OAuth

2019-01-11 Thread Hadoop QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16740891#comment-16740891
 ] 

Hadoop QA commented on HADOOP-15954:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m 
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
13m 12s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
21s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
16s{color} | {color:green} hadoop-tools/hadoop-azure: The patch generated 0 new 
+ 2 unchanged - 1 fixed = 2 total (was 3) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
13m 41s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m 
11s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
24s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 54m 33s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HADOOP-15954 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12954645/HADOOP-15954-011.patch
 |
| Optional Tests |  dupname  asflicense  compile  javac  javadoc  mvninstall  
mvnsite  unit  shadedclient  findbugs  checkstyle  |
| uname | Linux 6955c6a39e8e 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct 
5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / bf08f4a |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/15778/testReport/ |
| Max. process+thread count | 341 (vs. ulimit of 1) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/15778/console |
| Powered by | Apache Yetus 0.8.0   http://yetus.apache.org |


This

[jira] [Commented] (HADOOP-15954) ABFS: Enable owner and group conversion for MSI and login user using OAuth

2019-01-11 Thread Da Zhou (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16740851#comment-16740851
 ] 

Da Zhou commented on HADOOP-15954:
--

patch 010 doesn't apply to trunk due to recent commits.
Submitting patch 011.

> ABFS: Enable owner and group conversion for MSI and login user using OAuth
> --
>
> Key: HADOOP-15954
> URL: https://issues.apache.org/jira/browse/HADOOP-15954
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: junhua gu
>Assignee: Da Zhou
>Priority: Major
> Attachments: HADOOP-15954-001.patch, HADOOP-15954-002.patch, 
> HADOOP-15954-003.patch, HADOOP-15954-004.patch, HADOOP-15954-005.patch, 
> HADOOP-15954-006.patch, HADOOP-15954-007.patch, HADOOP-15954-008.patch, 
> HADOOP-15954-009.patch, HADOOP-15954-010.patch, HADOOP-15954-011.patch
>
>
> Add support for overwriting owner and group in set/get operations to be the 
> service principal id when OAuth is used. Add support for upn short name 
> format.
>  
> Add Standard Transformer for SharedKey / Service 
> Add interface provides an extensible model for customizing the acquisition of 
> Identity Transformer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15954) ABFS: Enable owner and group conversion for MSI and login user using OAuth

2019-01-10 Thread Hadoop QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16739961#comment-16739961
 ] 

Hadoop QA commented on HADOOP-15954:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m 
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
12m 45s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
20s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
16s{color} | {color:green} hadoop-tools/hadoop-azure: The patch generated 0 new 
+ 2 unchanged - 1 fixed = 2 total (was 3) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
13m 33s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m 
15s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
27s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 55m 24s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HADOOP-15954 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12954507/HADOOP-15954-010.patch
 |
| Optional Tests |  dupname  asflicense  compile  javac  javadoc  mvninstall  
mvnsite  unit  shadedclient  findbugs  checkstyle  |
| uname | Linux 971a1225892f 4.4.0-134-generic #160~14.04.1-Ubuntu SMP Fri Aug 
17 11:07:07 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 33c009a4 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/15772/testReport/ |
| Max. process+thread count | 306 (vs. ulimit of 1) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/15772/console |
| Powered by | Apache Yetus 0.8.0   http://yetus.apache.org |


This

[jira] [Commented] (HADOOP-15954) ABFS: Enable owner and group conversion for MSI and login user using OAuth

2019-01-10 Thread Hadoop QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16739956#comment-16739956
 ] 

Hadoop QA commented on HADOOP-15954:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m 
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
12m 46s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
22s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
17s{color} | {color:green} hadoop-tools/hadoop-azure: The patch generated 0 new 
+ 2 unchanged - 1 fixed = 2 total (was 3) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
13m 26s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m 
11s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
22s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 53m 57s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HADOOP-15954 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12954495/HADOOP-15954-009.patch
 |
| Optional Tests |  dupname  asflicense  compile  javac  javadoc  mvninstall  
mvnsite  unit  shadedclient  findbugs  checkstyle  |
| uname | Linux 3d4a5e7f2231 4.4.0-134-generic #160~14.04.1-Ubuntu SMP Fri Aug 
17 11:07:07 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 33c009a4 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/15771/testReport/ |
| Max. process+thread count | 341 (vs. ulimit of 1) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/15771/console |
| Powered by | Apache Yetus 0.8.0   http://yetus.apache.org |


This

[jira] [Commented] (HADOOP-15954) ABFS: Enable owner and group conversion for MSI and login user using OAuth

2019-01-10 Thread Da Zhou (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16739924#comment-16739924
 ] 

Da Zhou commented on HADOOP-15954:
--

Submitting patch 010. 
 *AbfsConfiguration:*
 Fixed, "org.apache.hadoop.fs.azurebfs.oauth2.DefaultSPIdentityTransformer" 
will be loaded and initialize if config is not set.

*DefaultSPIdentityTransformer:* 
 Removed the unnecessary changes for display name since this is not supported.
{quote}line 116 & 155: Should it be if 
(shouldReplaceOwnerAndGroup(replaceOwner) || (SUPER_USER.equals(replaceOwner) 
&& enableSuperUserReplacement))
{quote}
It is OK since A || (B && C) is the same as A || B&, for more complex 
condition I would add the parentheses for readability.

*AzureBlobFileSystem:*
 I think you mean "*AzureBlobFileSystemStore*", I've removed the unnecessary 
check for the null for "*identityTransformer*".

> ABFS: Enable owner and group conversion for MSI and login user using OAuth
> --
>
> Key: HADOOP-15954
> URL: https://issues.apache.org/jira/browse/HADOOP-15954
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: junhua gu
>Assignee: Da Zhou
>Priority: Major
> Attachments: HADOOP-15954-001.patch, HADOOP-15954-002.patch, 
> HADOOP-15954-003.patch, HADOOP-15954-004.patch, HADOOP-15954-005.patch, 
> HADOOP-15954-006.patch, HADOOP-15954-007.patch, HADOOP-15954-008.patch, 
> HADOOP-15954-009.patch
>
>
> Add support for overwriting owner and group in set/get operations to be the 
> service principal id when OAuth is used. Add support for upn short name 
> format.
>  
> Add Standard Transformer for SharedKey / Service 
> Add interface provides an extensible model for customizing the acquisition of 
> Identity Transformer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15954) ABFS: Enable owner and group conversion for MSI and login user using OAuth

2019-01-10 Thread Da Zhou (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16739910#comment-16739910
 ] 

Da Zhou commented on HADOOP-15954:
--

Thanks, will update it shortly.

> ABFS: Enable owner and group conversion for MSI and login user using OAuth
> --
>
> Key: HADOOP-15954
> URL: https://issues.apache.org/jira/browse/HADOOP-15954
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: junhua gu
>Assignee: Da Zhou
>Priority: Major
> Attachments: HADOOP-15954-001.patch, HADOOP-15954-002.patch, 
> HADOOP-15954-003.patch, HADOOP-15954-004.patch, HADOOP-15954-005.patch, 
> HADOOP-15954-006.patch, HADOOP-15954-007.patch, HADOOP-15954-008.patch, 
> HADOOP-15954-009.patch
>
>
> Add support for overwriting owner and group in set/get operations to be the 
> service principal id when OAuth is used. Add support for upn short name 
> format.
>  
> Add Standard Transformer for SharedKey / Service 
> Add interface provides an extensible model for customizing the acquisition of 
> Identity Transformer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15954) ABFS: Enable owner and group conversion for MSI and login user using OAuth

2019-01-10 Thread junhua gu (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16739890#comment-16739890
 ] 

junhua gu commented on HADOOP-15954:


*AbfsConfiguration*

Load DefaultSPIdentityTransfoermer when config is not been set in line 469.

*DefaultSPIdentityTransformer*

Need to remove the support for the faked service principal display name. Remove 
attribute servicePrincipalName and method replaceNameForSP, and change related 
function accordingly in line 55, 66, 82, 152, 153, 272, 280.

line 116 & 155: Should it be  if (shouldReplaceOwnerAndGroup(replaceOwner) || 
(SUPER_USER.equals(replaceOwner) && enableSuperUserReplacement))

*AzureBlobFileSystem*

No longer need to perform null check for the identity transformer, since there 
will always be a default implementation in line 536, 598, 618, 619, 682, 761, 
807.

 

 

 

> ABFS: Enable owner and group conversion for MSI and login user using OAuth
> --
>
> Key: HADOOP-15954
> URL: https://issues.apache.org/jira/browse/HADOOP-15954
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: junhua gu
>Assignee: Da Zhou
>Priority: Major
> Attachments: HADOOP-15954-001.patch, HADOOP-15954-002.patch, 
> HADOOP-15954-003.patch, HADOOP-15954-004.patch, HADOOP-15954-005.patch, 
> HADOOP-15954-006.patch, HADOOP-15954-007.patch, HADOOP-15954-008.patch, 
> HADOOP-15954-009.patch
>
>
> Add support for overwriting owner and group in set/get operations to be the 
> service principal id when OAuth is used. Add support for upn short name 
> format.
>  
> Add Standard Transformer for SharedKey / Service 
> Add interface provides an extensible model for customizing the acquisition of 
> Identity Transformer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15954) ABFS: Enable owner and group conversion for MSI and login user using OAuth

2019-01-10 Thread Hadoop QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16739881#comment-16739881
 ] 

Hadoop QA commented on HADOOP-15954:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m 
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
12m 49s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
21s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
16s{color} | {color:green} hadoop-tools/hadoop-azure: The patch generated 0 new 
+ 2 unchanged - 1 fixed = 2 total (was 3) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
13m 29s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m 
11s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
26s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 55m  6s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HADOOP-15954 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12954495/HADOOP-15954-009.patch
 |
| Optional Tests |  dupname  asflicense  compile  javac  javadoc  mvninstall  
mvnsite  unit  shadedclient  findbugs  checkstyle  |
| uname | Linux 53f417f4a874 4.4.0-134-generic #160~14.04.1-Ubuntu SMP Fri Aug 
17 11:07:07 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 33c009a4 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/15768/testReport/ |
| Max. process+thread count | 340 (vs. ulimit of 1) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/15768/console |
| Powered by | Apache Yetus 0.8.0   http://yetus.apache.org |


This

[jira] [Commented] (HADOOP-15954) ABFS: Enable owner and group conversion for MSI and login user using OAuth

2019-01-10 Thread Hadoop QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16739870#comment-16739870
 ] 

Hadoop QA commented on HADOOP-15954:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m 
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
13m  6s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
21s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
17s{color} | {color:green} hadoop-tools/hadoop-azure: The patch generated 0 new 
+ 2 unchanged - 1 fixed = 2 total (was 3) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
13m 34s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m 
11s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
25s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 54m  4s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HADOOP-15954 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12954493/HADOOP-15954-009.patch
 |
| Optional Tests |  dupname  asflicense  compile  javac  javadoc  mvninstall  
mvnsite  unit  shadedclient  findbugs  checkstyle  |
| uname | Linux be8613e85af5 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct 
5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 33c009a4 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/15767/testReport/ |
| Max. process+thread count | 307 (vs. ulimit of 1) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/15767/console |
| Powered by | Apache Yetus 0.8.0   http://yetus.apache.org |


This

[jira] [Commented] (HADOOP-15954) ABFS: Enable owner and group conversion for MSI and login user using OAuth

2019-01-10 Thread Da Zhou (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16739846#comment-16739846
 ] 

Da Zhou commented on HADOOP-15954:
--

Submitting 009-patch:
 To avoid breaking changes that influence current customers, the *$superuser* 
replacement has been added back.
 To make it more controllable by user, the replacement is moved to 
*DefaultSPIdentityTransformer*  and a new config option 
*_"fs.azure.default.identity.transformer.enable.superuser.replacement_*" has 
been introduced so this replacement can be disabled if user wants to, by 
default it is set as true.

Changes:
- Added back $superuser replacement
- Introduced config option  
*_"fs.azure.default.identity.transformer.enable.superuser.replacement_*" 
- Renamed "fs.azure.identity.transformer.class" to 
"fs.azure.identity.transformer.type" to keep the naming consistent.
- Updated testing_azure.md.

Tests:
All tests passed for XNS account (Oauth & sharedKey) and non-XNS account 
(SharedKey), the account used for test is located at US west.


> ABFS: Enable owner and group conversion for MSI and login user using OAuth
> --
>
> Key: HADOOP-15954
> URL: https://issues.apache.org/jira/browse/HADOOP-15954
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: junhua gu
>Assignee: Da Zhou
>Priority: Major
> Attachments: HADOOP-15954-001.patch, HADOOP-15954-002.patch, 
> HADOOP-15954-003.patch, HADOOP-15954-004.patch, HADOOP-15954-005.patch, 
> HADOOP-15954-006.patch, HADOOP-15954-007.patch, HADOOP-15954-008.patch, 
> HADOOP-15954-009.patch
>
>
> Add support for overwriting owner and group in set/get operations to be the 
> service principal id when OAuth is used. Add support for upn short name 
> format.
>  
> Add Standard Transformer for SharedKey / Service 
> Add interface provides an extensible model for customizing the acquisition of 
> Identity Transformer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15954) ABFS: Enable owner and group conversion for MSI and login user using OAuth

2019-01-09 Thread Hadoop QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16738832#comment-16738832
 ] 

Hadoop QA commented on HADOOP-15954:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
26s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m 
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
14m 15s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
20s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
17s{color} | {color:green} hadoop-tools/hadoop-azure: The patch generated 0 new 
+ 2 unchanged - 1 fixed = 2 total (was 3) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
14m  1s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m 
12s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
24s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 59m 42s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HADOOP-15954 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12954363/HADOOP-15954-008.patch
 |
| Optional Tests |  dupname  asflicense  compile  javac  javadoc  mvninstall  
mvnsite  unit  shadedclient  findbugs  checkstyle  |
| uname | Linux eb465ce5f79f 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct 
5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / f4617c6 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/15762/testReport/ |
| Max. process+thread count | 295 (vs. ulimit of 1) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/15762/console |
| Powered by | Apache Yetus 0.8.0   http://yetus.apache.org |


This

[jira] [Commented] (HADOOP-15954) ABFS: Enable owner and group conversion for MSI and login user using OAuth

2019-01-09 Thread Da Zhou (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16738764#comment-16738764
 ] 

Da Zhou commented on HADOOP-15954:
--

Submitted 008 patch:
- Revert the change for check on isSecurityEnabled(), the reason is even in 
non-secure env, it is possible that name conversion is needed (daemon service 
with MSI). User can still  disable the it through configuration files.

All tests passed:
non-XNS account, ShareKey: 
Tests run: 35, Failures: 0, Errors: 0, Skipped: 0
Tests run: 316, Failures: 0, Errors: 0, Skipped: 205
Tests run: 165, Failures: 0, Errors: 0, Skipped: 15

XNS account, sharedKey:
Tests run: 35, Failures: 0, Errors: 0, Skipped: 0
Tests run: 316, Failures: 0, Errors: 0, Skipped: 19
Tests run: 165, Failures: 0, Errors: 0, Skipped: 15

XNS account, OAuth:
Tests run: 35, Failures: 0, Errors: 0, Skipped: 0
Tests run: 316, Failures: 0, Errors: 0, Skipped: 20
Tests run: 165, Failures: 0, Errors: 0, Skipped: 21

> ABFS: Enable owner and group conversion for MSI and login user using OAuth
> --
>
> Key: HADOOP-15954
> URL: https://issues.apache.org/jira/browse/HADOOP-15954
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: junhua gu
>Assignee: Da Zhou
>Priority: Major
> Attachments: HADOOP-15954-001.patch, HADOOP-15954-002.patch, 
> HADOOP-15954-003.patch, HADOOP-15954-004.patch, HADOOP-15954-005.patch, 
> HADOOP-15954-006.patch, HADOOP-15954-007.patch, HADOOP-15954-008.patch
>
>
> Add support for overwriting owner and group in set/get operations to be the 
> service principal id when OAuth is used. Add support for upn short name 
> format.
>  
> Add Standard Transformer for SharedKey / Service 
> Add interface provides an extensible model for customizing the acquisition of 
> Identity Transformer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15954) ABFS: Enable owner and group conversion for MSI and login user using OAuth

2019-01-09 Thread Da Zhou (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16738735#comment-16738735
 ] 

Da Zhou commented on HADOOP-15954:
--

[~ste...@apache.org], good point, I would like to keep this in mind and see how 
is things going. Something like would benefit us a lot.


> ABFS: Enable owner and group conversion for MSI and login user using OAuth
> --
>
> Key: HADOOP-15954
> URL: https://issues.apache.org/jira/browse/HADOOP-15954
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: junhua gu
>Assignee: Da Zhou
>Priority: Major
> Attachments: HADOOP-15954-001.patch, HADOOP-15954-002.patch, 
> HADOOP-15954-003.patch, HADOOP-15954-004.patch, HADOOP-15954-005.patch, 
> HADOOP-15954-006.patch, HADOOP-15954-007.patch
>
>
> Add support for overwriting owner and group in set/get operations to be the 
> service principal id when OAuth is used. Add support for upn short name 
> format.
>  
> Add Standard Transformer for SharedKey / Service 
> Add interface provides an extensible model for customizing the acquisition of 
> Identity Transformer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15954) ABFS: Enable owner and group conversion for MSI and login user using OAuth

2019-01-09 Thread Steve Loughran (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16738207#comment-16738207
 ] 

Steve Loughran commented on HADOOP-15954:
-

I've been working on Delegation Tokens for S3A and thinking about this

Should whatever plugin that ABFS (& other stores) use for DT binding hold the 
right to declare the username & group?  That way, if someone is logged in with 
AD, it can get user & group from that, and if a DT was issued from the logged 
in user, the user & group could be added to the DT and unmarshalled at the far 
end?

I don't think that'll directly impact this patch, but it is ~related...if 
something like that went it, it'd be an evolution

> ABFS: Enable owner and group conversion for MSI and login user using OAuth
> --
>
> Key: HADOOP-15954
> URL: https://issues.apache.org/jira/browse/HADOOP-15954
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: junhua gu
>Assignee: Da Zhou
>Priority: Major
> Attachments: HADOOP-15954-001.patch, HADOOP-15954-002.patch, 
> HADOOP-15954-003.patch, HADOOP-15954-004.patch, HADOOP-15954-005.patch, 
> HADOOP-15954-006.patch, HADOOP-15954-007.patch
>
>
> Add support for overwriting owner and group in set/get operations to be the 
> service principal id when OAuth is used. Add support for upn short name 
> format.
>  
> Add Standard Transformer for SharedKey / Service 
> Add interface provides an extensible model for customizing the acquisition of 
> Identity Transformer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15954) ABFS: Enable owner and group conversion for MSI and login user using OAuth

2019-01-07 Thread Hadoop QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16736330#comment-16736330
 ] 

Hadoop QA commented on HADOOP-15954:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m 
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
13m 11s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
20s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
16s{color} | {color:green} hadoop-tools/hadoop-azure: The patch generated 0 new 
+ 2 unchanged - 1 fixed = 2 total (was 3) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
13m 28s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m  
9s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
25s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 53m 52s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HADOOP-15954 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12954042/HADOOP-15954-007.patch
 |
| Optional Tests |  dupname  asflicense  compile  javac  javadoc  mvninstall  
mvnsite  unit  shadedclient  findbugs  checkstyle  |
| uname | Linux 452bc201ae5b 4.4.0-134-generic #160~14.04.1-Ubuntu SMP Fri Aug 
17 11:07:07 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 06279ec |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/15743/testReport/ |
| Max. process+thread count | 341 (vs. ulimit of 1) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/15743/console |
| Powered by | Apache Yetus 0.8.0   http://yetus.apache.org |


This

[jira] [Commented] (HADOOP-15954) ABFS: Enable owner and group conversion for MSI and login user using OAuth

2019-01-07 Thread Hadoop QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16736308#comment-16736308
 ] 

Hadoop QA commented on HADOOP-15954:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
27s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m 
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
12m 54s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
20s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
15s{color} | {color:green} hadoop-tools/hadoop-azure: The patch generated 0 new 
+ 2 unchanged - 1 fixed = 2 total (was 3) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
13m 31s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m 
13s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
24s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 57m  3s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HADOOP-15954 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12954038/HADOOP-15954-007.patch
 |
| Optional Tests |  dupname  asflicense  compile  javac  javadoc  mvninstall  
mvnsite  unit  shadedclient  findbugs  checkstyle  |
| uname | Linux 5ef785ddb5a8 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct 
5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 06279ec |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/15742/testReport/ |
| Max. process+thread count | 294 (vs. ulimit of 1) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/15742/console |
| Powered by | Apache Yetus 0.8.0   http://yetus.apache.org |


This

[jira] [Commented] (HADOOP-15954) ABFS: Enable owner and group conversion for MSI and login user using OAuth

2019-01-07 Thread Da Zhou (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16736247#comment-16736247
 ] 

Da Zhou commented on HADOOP-15954:
--

Thanks for the review.
Attaching 007 patch:
- Removed the unnecessary replacement for $superuser
- updated the DefaultSPIdentityTransformer so it is enabled only when 
isSecurityEnabled is true.
- replaced the log.error() with log.warn() in 
DefaultSPIdentityTransformer.initialize() when fetching name/group from 
userGroupInformation.
-  I see in KerbroseName.java "Locale.ENGLISH" is being used, updated 
toLowerCase() to toLowerCase(Locale.ENGLISH).

All tests passed:
non-XNS account, ShareKey: 
Tests run: 35, Failures: 0, Errors: 0, Skipped: 0
Tests run: 316, Failures: 0, Errors: 0, Skipped: 205
Tests run: 165, Failures: 0, Errors: 0, Skipped: 15

XNS account, sharedKey:
Tests run: 35, Failures: 0, Errors: 0, Skipped: 0
Tests run: 316, Failures: 0, Errors: 0, Skipped: 19
Tests run: 165, Failures: 0, Errors: 0, Skipped: 15

XNS account, OAuth:
Tests run: 35, Failures: 0, Errors: 0, Skipped: 0
Tests run: 316, Failures: 0, Errors: 0, Skipped: 20
Tests run: 165, Failures: 0, Errors: 0, Skipped: 21

> ABFS: Enable owner and group conversion for MSI and login user using OAuth
> --
>
> Key: HADOOP-15954
> URL: https://issues.apache.org/jira/browse/HADOOP-15954
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: junhua gu
>Assignee: Da Zhou
>Priority: Major
> Attachments: HADOOP-15954-001.patch, HADOOP-15954-002.patch, 
> HADOOP-15954-003.patch, HADOOP-15954-004.patch, HADOOP-15954-005.patch, 
> HADOOP-15954-006.patch, HADOOP-15954-007.patch
>
>
> Add support for overwriting owner and group in set/get operations to be the 
> service principal id when OAuth is used. Add support for upn short name 
> format.
>  
> Add Standard Transformer for SharedKey / Service 
> Add interface provides an extensible model for customizing the acquisition of 
> Identity Transformer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15954) ABFS: Enable owner and group conversion for MSI and login user using OAuth

2018-12-17 Thread Steve Loughran (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16722899#comment-16722899
 ] 

Steve Loughran commented on HADOOP-15954:
-

h3. {{DefaultSPIdentityTransformer.transformAclEntries}}


I don't see this feature being needed at all when !isSecurityEnabled, but on my 
reading of the code it's going to log at error every time initialize() is 
called. This isn't appropriate there.

# only worry about name extraction when running secure
# Log @ warn

I'd worry about the logs being full of these error messages in any long-lived 
service,
Spark, Hive LLAP, where FS instances are not just created, they are destroyed 
afte work is done (especially LLAP). Is there a way to minimise the logging?


h3. {{getShortName}}

Is the case conversion going to work in all locales, or should the locale for 
the toLowerCase() call be set to LOCALE_EN? I ask as I don't know how 
AD/kerberos realms
with I in their name get converted in Turkey, but I suspect it's not what you 
want
across a global system.

Elsewhere (/HADOOP-15996) we're looking at how to handle more complex names,
e.g. cross realm problems and users who have an @ in their short name.
Is this code going to handle that? As a plug-in mechanism is underway, 
getting involved in that/designing the code for it (how?) is wise.


+ general, minor: Use the size of the incoming list to set the size of the 
output ArrayList; saves reallocation & GC


> ABFS: Enable owner and group conversion for MSI and login user using OAuth
> --
>
> Key: HADOOP-15954
> URL: https://issues.apache.org/jira/browse/HADOOP-15954
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: junhua gu
>Assignee: Da Zhou
>Priority: Major
> Attachments: HADOOP-15954-001.patch, HADOOP-15954-002.patch, 
> HADOOP-15954-003.patch, HADOOP-15954-004.patch, HADOOP-15954-005.patch, 
> HADOOP-15954-006.patch
>
>
> Add support for overwriting owner and group in set/get operations to be the 
> service principal id when OAuth is used. Add support for upn short name 
> format.
>  
> Add Standard Transformer for SharedKey / Service 
> Add interface provides an extensible model for customizing the acquisition of 
> Identity Transformer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15954) ABFS: Enable owner and group conversion for MSI and login user using OAuth

2018-12-11 Thread Hadoop QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718225#comment-16718225
 ] 

Hadoop QA commented on HADOOP-15954:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m 
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
12m 59s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
21s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
16s{color} | {color:green} hadoop-tools/hadoop-azure: The patch generated 0 new 
+ 2 unchanged - 1 fixed = 2 total (was 3) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
13m 45s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m 
11s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
28s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 55m 11s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HADOOP-15954 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12951428/HADOOP-15954-006.patch
 |
| Optional Tests |  dupname  asflicense  compile  javac  javadoc  mvninstall  
mvnsite  unit  shadedclient  findbugs  checkstyle  |
| uname | Linux e75b10aa505d 4.4.0-134-generic #160~14.04.1-Ubuntu SMP Fri Aug 
17 11:07:07 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / fb55e52 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/15638/testReport/ |
| Max. process+thread count | 307 (vs. ulimit of 1) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/15638/console |
| Powered by | Apache Yetus 0.8.0   http://yetus.apache.org |


This

[jira] [Commented] (HADOOP-15954) ABFS: Enable owner and group conversion for MSI and login user using OAuth

2018-12-11 Thread Hadoop QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718011#comment-16718011
 ] 

Hadoop QA commented on HADOOP-15954:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m 
 5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
11m 39s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
24s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
24s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 17s{color} | {color:orange} hadoop-tools/hadoop-azure: The patch generated 2 
new + 2 unchanged - 1 fixed = 4 total (was 3) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
26s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 3 line(s) that end in whitespace. Use git 
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply 
{color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
12m 51s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
45s{color} | {color:red} hadoop-tools/hadoop-azure generated 1 new + 0 
unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
33s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  1m 59s{color} 
| {color:red} hadoop-azure in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
22s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 56m 43s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-tools/hadoop-azure |
|  |  Unread field:DefaultSPIdentityTransformer.java:[line 91] |
| Failed junit tests | hadoop.fs.azure.TestClientThrottlingAnalyzer |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HADOOP-15954 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12951416/HADOOP-15954-005.patch
 |
| Optional Tests |  dupname  asflicense  compile  javac  javadoc  mvninstall  
mvnsite  unit  shadedclient  findbugs  checkstyle  |
| uname | Linux 1460eabd9795 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 881230d |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| checkstyle |

[jira] [Commented] (HADOOP-15954) ABFS: Enable owner and group conversion for MSI and login user using OAuth

2018-12-11 Thread Da Zhou (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16717959#comment-16717959
 ] 

Da Zhou commented on HADOOP-15954:
--

Attached 005 patch, all existed tests passed, it is tricky to add new tests for 
this feature, still working on that.
 - Modified the initialize() method of transformer class, so that FS can pass 
its cached user information to the transformer.
 - updated null check in "AbfsConfiguration.getIdentity transfomer()"
 - Addressed comments from [~tmarquardt] except 1-b:
 transformUserName() and transformGroupName() are used to make the interface 
simple, otherwise we need to create a new class for represent for username and 
group name.
 - Renamed ""fs.azure.identity.transformer.*type*" to 
""fs.azure.identity.transformer.*class*"

All existed tests passed:
 non-XNS account, ShareKey: 
 Tests run: 35, Failures: 0, Errors: 0, Skipped: 0
 Tests run: 316, Failures: 0, Errors: 0, Skipped: 205
 Tests run: 165, Failures: 0, Errors: 0, Skipped: 15

XNS account, sharedKey:
 Tests run: 35, Failures: 0, Errors: 0, Skipped: 0
 Tests run: 316, Failures: 0, Errors: 0, Skipped: 19
 Tests run: 165, Failures: 0, Errors: 0, Skipped: 15

XNS account, OAuth:
 Tests run: 35, Failures: 0, Errors: 0, Skipped: 0
 Tests run: 316, Failures: 0, Errors: 0, Skipped: 20
 Tests run: 165, Failures: 0, Errors: 0, Skipped: 21

> ABFS: Enable owner and group conversion for MSI and login user using OAuth
> --
>
> Key: HADOOP-15954
> URL: https://issues.apache.org/jira/browse/HADOOP-15954
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: junhua gu
>Assignee: Da Zhou
>Priority: Major
> Attachments: HADOOP-15954-001.patch, HADOOP-15954-002.patch, 
> HADOOP-15954-003.patch, HADOOP-15954-004.patch, HADOOP-15954-005.patch
>
>
> Add support for overwriting owner and group in set/get operations to be the 
> service principal id when OAuth is used. Add support for upn short name 
> format.
>  
> Add Standard Transformer for SharedKey / Service 
> Add interface provides an extensible model for customizing the acquisition of 
> Identity Transformer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15954) ABFS: Enable owner and group conversion for MSI and login user using OAuth

2018-12-07 Thread Thomas Marquardt (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713397#comment-16713397
 ] 

Thomas Marquardt commented on HADOOP-15954:
---

I have a few ideas about how we should design this feature:

1) There should be a public interface named CustomIdentityTransformer in the 
org.apache.hadoop.fs.azurebfs.extensions package.

    a) This class is already implemented in patch 004, but the name should be 
changed from CustomIdentityTransformerAdaptee to CustomIdentityTransformer.

    b) transformUserName and transformGroupName should be combined into a 
single function named transformOwner if possible. Note that these methods are 
invoked to transform the input of a FileSystem.setOwner call.

    c) All implementations of identity transformer will implement this 
interface, including the default implementation.

2) The AbfsConfiguration.getIdentityTransformer method should do the following:

    a) If "*fs.azure.identity.transformer.type*" is not set, return immediately 
without doing any additional work.

    b) If "*fs.azure.identity.transformer.type*" is set:

        i) Load the class.

        ii) Create a new instance.

        iii) Call initialize and pass in the configuration and account name.

        iv) Return the new initialized instance of CustomIdentityTransformer.

3) Remove the configuration "*fs.azure.account.identity.type*".  It is 
superfluous.

4) Remove the enum TransformerType.  It is superfluous since the 
"*fs.azure.identity.transformer.type*" will define the type of identity 
transformer used.

5) Remove the class 
org.apache.hadoop.fs.azurebfs.oauth.CustomIdentityTransformerAdapter.

    a) The Adapter Design model is not appropriate here.  The Adapter Design 
model is used when you have a public interface, but there are external 
component that implement a different interface.  In our case there is no need 
to have an adapter, because the external component can implement our new public 
interface directly.

6) Provide a default implementation named DefaultIdentityTransformer in the 
org.apache.hadoop.fs.azurebfs.oauth2 package.

    a) This class is partly implemented, but:

        i) The name should be changed from StandardSPIdentityTransformer to 
DefaultIdentityTransformer.

        ii) DefaultIdentityTransformer should implement 
CustomIdentityTransformer. The IdentityTransformer code should be moved into 
DefaultIdentityTransformer.

 

7) Configuration used by DefaultIdentityTransformer should be prefixed with 
“fs.azure.default.identity.transformer”:

    a) “*fs.azure.default.identity.transformer.enable.short.name*”  – user 
principal names (UPNs) have the format “\{alias}@\{domain}”.  If true, only 
\{alias} is included when a UPN would otherwise appear in the output of APIs 
like getFileStatus, getOwner, getAclStatus, etc. 

    b) “*fs.azure.default.identity.transformer.domain.name*” – If the domain 
name is specified and “fs.azure.default.identity.transformer.enable.short.name” 
is true, then the \{alias} part of a UPN can be specified as input to APIs like 
setOwner and setAcl and it will be transformed to a UPN by appending @ and the 
domain specified by this configuration property.

    c) 
“*fs.azure.default.identity.transformer.service.principal.substitution.list*” – 
a comma separated list of names to be replaced with the service principal ID 
specified by “*fs.azure.default.identity.transformer.service.principal.id*”.  
This substitution occurs when setOwner, setAcl, modifyAclEntries, or 
removeAclEntries are invoked with identities contained in the substitution list.

    d) “*fs.azure.default.identity.transformer.service.principal.id*” – an 
Azure Active Directory object ID used as the replacement for names contained in 
the list specified by 
“*fs.azure.default.identity.transformer.service.principal.substitution.list*”.

    e) The patch 004 properties defined in ConfigurationKeys should be replaced 
with the above.  Please include descriptions.

8) Remove the class org.apache.hadoop.fs.azurebfs.oauth.IdentityTransformer.

9) Remove DEFAULT_ENABLE_SHORT_NAME from FileSystemConfigurations.  It can be 
moved to DefaultIdentityTransformer if it is needed.

10) I don’t see any reason for IdentityTransformerException to exist, since it 
does not have any implementation.  If it truly cannot be removed, I would 
rename it to *IdentityTransformerInitializationException*.

11) Update all the places in AzureBlobFileSystemStore where the 
CustomIdentityTransformer instance is invoked to check for null, and only 
perform the transformation if non-null.

12) By default, there will be no transformation. A customer can set 
fs.azure.identity.transformer.type to the default or a custom implementation as 
needed.

13) We should have tests for the implementation of DefaultIdentityTransformer.

 

> ABFS: Enable owner and group conversion for MSI and login user

[jira] [Commented] (HADOOP-15954) ABFS: Enable owner and group conversion for MSI and login user using OAuth

2018-12-04 Thread Steve Loughran (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16709038#comment-16709038
 ] 

Steve Loughran commented on HADOOP-15954:
-

Things don't compile; I'll leave you to work out what went wrong there.

Qucik review of some parts of the code:

* Some minimal docs will be needed for this. Even if currently there is one 
single user, there may be more in future, and all of us will end up trying to 
debug a stack trace related to it at some point
* It's critical that the "owner" of the FS is used over the 
UGI.getCurrentUser(); the owner is cached in {{AzureBlobFileSystem.initialize}} 
and can be retrieved later.
Recommend: pass it in as a constructor to the transformer. That'll make testing 
in a miniKDC cluster viable.


h3. getIdentityTransformer

switch to using Preconditions.checkArgument/checkNotNull for validation

In the final catch Exception & wrap, include toString value of caught ex in the 
new exception message.

> ABFS: Enable owner and group conversion for MSI and login user using OAuth
> --
>
> Key: HADOOP-15954
> URL: https://issues.apache.org/jira/browse/HADOOP-15954
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: junhua gu
>Assignee: junhua gu
>Priority: Major
> Attachments: HADOOP-15954-001.patch, HADOOP-15954-002.patch, 
> HADOOP-15954-003.patch, HADOOP-15954-004.patch
>
>
> Add support for overwriting owner and group in set/get operations to be the 
> service principal id when OAuth is used. Add support for upn short name 
> format.
>  
> Add Standard Transformer for SharedKey / Service 
> Add interface provides an extensible model for customizing the acquisition of 
> Identity Transformer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15954) ABFS: Enable owner and group conversion for MSI and login user using OAuth

2018-12-03 Thread Hadoop QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16708052#comment-16708052
 ] 

Hadoop QA commented on HADOOP-15954:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  8m 
45s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m 
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
11m 23s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
18s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
15s{color} | {color:red} hadoop-azure in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  0m 
16s{color} | {color:red} hadoop-azure in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  0m 16s{color} 
| {color:red} hadoop-azure in the patch failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
14s{color} | {color:green} hadoop-tools/hadoop-azure: The patch generated 0 new 
+ 2 unchanged - 1 fixed = 2 total (was 3) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red}  0m 
17s{color} | {color:red} hadoop-azure in the patch failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
11m 33s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
17s{color} | {color:red} hadoop-azure in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  0m 16s{color} 
| {color:red} hadoop-azure in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
24s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 55m 46s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HADOOP-15954 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12950474/HADOOP-15954-004.patch
 |
| Optional Tests |  dupname  asflicense  compile  javac  javadoc  mvninstall  
mvnsite  unit  shadedclient  findbugs  checkstyle  |
| uname | Linux ade3b094a896 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / de42555 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| mvninstall | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/15602/artifact/out/patch-mvninstall-hadoop-tools_hadoop-azure.txt
 |
| compile | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/15602/artifact/out/patch-compile-hadoop-tools_hadoop-azure.txt
 |
| javac |

[jira] [Commented] (HADOOP-15954) ABFS: Enable owner and group conversion for MSI and login user using OAuth

2018-12-03 Thread junhua gu (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16708014#comment-16708014
 ] 

junhua gu commented on HADOOP-15954:


HADOOP-15954-004
 # Didn't move *skipTransFormation* to "AzureBlobFileSystemStore" since 
initialization for "IdentityTransformer" is always required. We can add a 
default Transformer do "input=output" if we really want to do it.**
 # Fix IdentityTransformerException and checkstyle issue.

> ABFS: Enable owner and group conversion for MSI and login user using OAuth
> --
>
> Key: HADOOP-15954
> URL: https://issues.apache.org/jira/browse/HADOOP-15954
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: junhua gu
>Assignee: junhua gu
>Priority: Major
> Attachments: HADOOP-15954-001.patch, HADOOP-15954-002.patch, 
> HADOOP-15954-003.patch, HADOOP-15954-004.patch
>
>
> Add support for overwriting owner and group in set/get operations to be the 
> service principal id when OAuth is used. Add support for upn short name 
> format.
>  
> Add Standard Transformer for SharedKey / Service 
> Add interface provides an extensible model for customizing the acquisition of 
> Identity Transformer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15954) ABFS: Enable owner and group conversion for MSI and login user using OAuth

2018-12-03 Thread Hadoop QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16707688#comment-16707688
 ] 

Hadoop QA commented on HADOOP-15954:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m 
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
12m 54s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
22s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
17s{color} | {color:green} hadoop-tools/hadoop-azure: The patch generated 0 new 
+ 2 unchanged - 1 fixed = 2 total (was 3) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
13m 42s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m 
11s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
24s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 55m  9s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HADOOP-15954 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12950057/HADOOP-15954-003.patch
 |
| Optional Tests |  dupname  asflicense  compile  javac  javadoc  mvninstall  
mvnsite  unit  shadedclient  findbugs  checkstyle  |
| uname | Linux 1809a567c279 4.4.0-134-generic #160~14.04.1-Ubuntu SMP Fri Aug 
17 11:07:07 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / fb10803 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/15598/testReport/ |
| Max. process+thread count | 339 (vs. ulimit of 1) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/15598/console |
| Powered by | Apache Yetus 0.8.0   http://yetus.apache.org |


This

[jira] [Commented] (HADOOP-15954) ABFS: Enable owner and group conversion for MSI and login user using OAuth

2018-12-03 Thread Da Zhou (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16707633#comment-16707633
 ] 

Da Zhou commented on HADOOP-15954:
--

*IdentityTransformerExceptio*n: line 33
input parameter "cause" is declared but not used.

Flag "*skipTransFormation*" is defined and used in 
"StandardSPIdentityTransformer", I think it is better to add it to 
"AzureBlobFileSystemStore" to avoid the unnecessary initialization for 
"IdentityTransformer".

There are many checkstyle violations, so I'm submitting the patch to get the 
Yetus report then you can fix them in another patch.

Tests: Tested locally, all existing tests passed. But we need new tests for 
patch like this.

> ABFS: Enable owner and group conversion for MSI and login user using OAuth
> --
>
> Key: HADOOP-15954
> URL: https://issues.apache.org/jira/browse/HADOOP-15954
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: junhua gu
>Assignee: junhua gu
>Priority: Major
> Attachments: HADOOP-15954-001.patch, HADOOP-15954-002.patch, 
> HADOOP-15954-003.patch
>
>
> Add support for overwriting owner and group in set/get operations to be the 
> service principal id when OAuth is used. Add support for upn short name 
> format.
>  
> Add Standard Transformer for SharedKey / Service 
> Add interface provides an extensible model for customizing the acquisition of 
> Identity Transformer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15954) ABFS: Enable owner and group conversion for MSI and login user using OAuth

2018-11-29 Thread junhua gu (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16703653#comment-16703653
 ] 

junhua gu commented on HADOOP-15954:


HADOOP-15954-003

Add support for set owner, group and acls with short name.

> ABFS: Enable owner and group conversion for MSI and login user using OAuth
> --
>
> Key: HADOOP-15954
> URL: https://issues.apache.org/jira/browse/HADOOP-15954
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: junhua gu
>Assignee: junhua gu
>Priority: Major
> Attachments: HADOOP-15954-001.patch, HADOOP-15954-002.patch, 
> HADOOP-15954-003.patch
>
>
> Add support for overwriting owner and group in set/get operations to be the 
> service principal id when OAuth is used. Add support for upn short name 
> format.
>  
> Add Standard Transformer for SharedKey / Service 
> Add interface provides an extensible model for customizing the acquisition of 
> Identity Transformer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15954) ABFS: Enable owner and group conversion for MSI and login user using OAuth

2018-11-28 Thread Da Zhou (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16702660#comment-16702660
 ] 

Da Zhou commented on HADOOP-15954:
--

Line 599,600: should log effectiveOwner and effectiveGroup.

 

> ABFS: Enable owner and group conversion for MSI and login user using OAuth
> --
>
> Key: HADOOP-15954
> URL: https://issues.apache.org/jira/browse/HADOOP-15954
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: junhua gu
>Assignee: junhua gu
>Priority: Major
> Attachments: HADOOP-15954-001.patch, HADOOP-15954-002.patch
>
>
> Add support for overwriting owner and group in set/get operations to be the 
> service principal id when OAuth is used. Add support for upn short name 
> format.
>  
> Add Standard Transformer for SharedKey / Service 
> Add interface provides an extensible model for customizing the acquisition of 
> Identity Transformer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15954) ABFS: Enable owner and group conversion for MSI and login user using OAuth

2018-11-28 Thread junhua gu (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16702210#comment-16702210
 ] 

junhua gu commented on HADOOP-15954:


HADOOP-15954-001 is not the final version, do not complete.

> ABFS: Enable owner and group conversion for MSI and login user using OAuth
> --
>
> Key: HADOOP-15954
> URL: https://issues.apache.org/jira/browse/HADOOP-15954
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: junhua gu
>Priority: Major
> Attachments: HADOOP-15954-001.patch
>
>
> Add support for overwriting owner and group in set/get operations to be the 
> service principal id when OAuth is used. Add support for upn short name 
> format.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

50 matches

Mail list logo