[jira] [Commented] (HADOOP-15872) ABFS: Update to target 2018-11-09 REST version for ADLS Gen 2
[ https://issues.apache.org/jira/browse/HADOOP-15872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16697434#comment-16697434 ] Steve Loughran commented on HADOOP-15872: - sorry, typo. the ABLS one. And I've managed to lose the console with the logs I just got some 3-way merge warnings when I merged the files in, but didn't see any recent history on trunk as I did it (and this is the separate "write-access-to-ASF" source tree as opposed to the "where I break things in my IDE", so it was in sync. Nothing important, it just struct me as unexpected > ABFS: Update to target 2018-11-09 REST version for ADLS Gen 2 > - > > Key: HADOOP-15872 > URL: https://issues.apache.org/jira/browse/HADOOP-15872 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Affects Versions: 3.2.0 >Reporter: Thomas Marquardt >Assignee: junhua gu >Priority: Major > Fix For: 3.2.1 > > Attachments: HADOOP-15872-001.patch, HADOOP-15872-002.patch, > HADOOP-15872-003.patch, HADOOP-15872-004.patch, HADOOP-15872-005.patch > > > This update to the latest REST version (2018-11-09) will make the following > changes to the ABFS driver: > 1) The ABFS implementation of getFileStatus currently requires read > permission. According to HDFS permissions guide, it should only require > execute on the parent folders (traversal access). A new REST API has been > introduced in REST version "2018-11-09" of ADLS Gen 2 to fix this problem. > 2) The new "2018-11-09" REST version introduces support to i) automatically > translate UPNs to OIDs when setting the owner, owning group, or ACL and ii) > optionally translate OIDs to UPNs in the responses when getting the owner, > owning group, or ACL. Configuration will be introduced to optionally > translate OIDs to UPNs in the responses. Since translation has a performance > impact, the default will be to perform no translation and return the OIDs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15872) ABFS: Update to target 2018-11-09 REST version for ADLS Gen 2
[ https://issues.apache.org/jira/browse/HADOOP-15872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16697422#comment-16697422 ] Da Zhou commented on HADOOP-15872: -- Thank you Steve. The branch I'm working on is Hadoop trunk, since it passed the Hadoop QA check, I guess there shouldn't be merge issue? what branch is the AWS one you are referring to? > ABFS: Update to target 2018-11-09 REST version for ADLS Gen 2 > - > > Key: HADOOP-15872 > URL: https://issues.apache.org/jira/browse/HADOOP-15872 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Affects Versions: 3.2.0 >Reporter: Thomas Marquardt >Assignee: junhua gu >Priority: Major > Fix For: 3.2.1 > > Attachments: HADOOP-15872-001.patch, HADOOP-15872-002.patch, > HADOOP-15872-003.patch, HADOOP-15872-004.patch, HADOOP-15872-005.patch > > > This update to the latest REST version (2018-11-09) will make the following > changes to the ABFS driver: > 1) The ABFS implementation of getFileStatus currently requires read > permission. According to HDFS permissions guide, it should only require > execute on the parent folders (traversal access). A new REST API has been > introduced in REST version "2018-11-09" of ADLS Gen 2 to fix this problem. > 2) The new "2018-11-09" REST version introduces support to i) automatically > translate UPNs to OIDs when setting the owner, owning group, or ACL and ii) > optionally translate OIDs to UPNs in the responses when getting the owner, > owning group, or ACL. Configuration will be introduced to optionally > translate OIDs to UPNs in the responses. Since translation has a performance > impact, the default will be to perform no translation and return the OIDs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15872) ABFS: Update to target 2018-11-09 REST version for ADLS Gen 2
[ https://issues.apache.org/jira/browse/HADOOP-15872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16697278#comment-16697278 ] Hudson commented on HADOOP-15872: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15500 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/15500/]) HADOOP-15872. ABFS: Update to target 2018-11-09 REST version for ADLS (stevel: rev a8302e398c2ca4b0deff062c0e921053351b688e) * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsRestOperationType.java * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsAclHelper.java * (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFilesystemAcl.java * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/FileSystemConfigurations.java * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/AbfsHttpConstants.java * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/ConfigurationKeys.java * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/HttpQueryParams.java * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java * (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemAuthorization.java > ABFS: Update to target 2018-11-09 REST version for ADLS Gen 2 > - > > Key: HADOOP-15872 > URL: https://issues.apache.org/jira/browse/HADOOP-15872 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Affects Versions: 3.2.0 >Reporter: Thomas Marquardt >Assignee: junhua gu >Priority: Major > Fix For: 3.2.1 > > Attachments: HADOOP-15872-001.patch, HADOOP-15872-002.patch, > HADOOP-15872-003.patch, HADOOP-15872-004.patch, HADOOP-15872-005.patch > > > This update to the latest REST version (2018-11-09) will make the following > changes to the ABFS driver: > 1) The ABFS implementation of getFileStatus currently requires read > permission. According to HDFS permissions guide, it should only require > execute on the parent folders (traversal access). A new REST API has been > introduced in REST version "2018-11-09" of ADLS Gen 2 to fix this problem. > 2) The new "2018-11-09" REST version introduces support to i) automatically > translate UPNs to OIDs when setting the owner, owning group, or ACL and ii) > optionally translate OIDs to UPNs in the responses when getting the owner, > owning group, or ACL. Configuration will be introduced to optionally > translate OIDs to UPNs in the responses. Since translation has a performance > impact, the default will be to perform no translation and return the OIDs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15872) ABFS: Update to target 2018-11-09 REST version for ADLS Gen 2
[
https://issues.apache.org/jira/browse/HADOOP-15872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16696254#comment-16696254
]
Hadoop QA commented on HADOOP-15872:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 16s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 31s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
17s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 56m 18s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HADOOP-15872 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12949222/HADOOP-15872-005.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 4bc28bf221c0 4.4.0-134-generic #160~14.04.1-Ubuntu SMP Fri Aug
17 11:07:07 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / e223a79 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15552/testReport/ |
| Max. process+thread count | 310 (vs. ulimit of 1) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15552/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> ABFS: Update to target 2018-11-09 REST version for ADLS Gen 2
> -
>
> Key: HADOOP-15872
>
[jira] [Commented] (HADOOP-15872) ABFS: Update to target 2018-11-09 REST version for ADLS Gen 2
[ https://issues.apache.org/jira/browse/HADOOP-15872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16696216#comment-16696216 ] Da Zhou commented on HADOOP-15872: -- Attached 005 patch, fixed the checkstyle issue. Yes it is ready. *HNS account, using SharedKey* Tests run: 35, Failures: 0, Errors: 0, Skipped: 0 Tests run: 316, Failures: 0, Errors: 0, Skipped: 19 Tests run: 165, Failures: 0, Errors: 0, Skipped: 15 *HNS account, using OAuth* Tests run: 35, Failures: 0, Errors: 0, Skipped: 0 Tests run: 316, Failures: 0, Errors: 0, Skipped: 20 Tests run: 165, Failures: 0, Errors: 0, Skipped: 21 *Non-HNS account, using SharedKey:* Tests run: 35, Failures: 0, Errors: 0, Skipped: 0 Tests run: 316, Failures: 0, Errors: 0, Skipped: 205 Tests run: 165, Failures: 0, Errors: 0, Skipped: 15 > ABFS: Update to target 2018-11-09 REST version for ADLS Gen 2 > - > > Key: HADOOP-15872 > URL: https://issues.apache.org/jira/browse/HADOOP-15872 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Affects Versions: 3.2.0 >Reporter: Thomas Marquardt >Assignee: junhua gu >Priority: Major > Attachments: HADOOP-15872-001.patch, HADOOP-15872-002.patch, > HADOOP-15872-003.patch, HADOOP-15872-004.patch, HADOOP-15872-005.patch > > > This update to the latest REST version (2018-11-09) will make the following > changes to the ABFS driver: > 1) The ABFS implementation of getFileStatus currently requires read > permission. According to HDFS permissions guide, it should only require > execute on the parent folders (traversal access). A new REST API has been > introduced in REST version "2018-11-09" of ADLS Gen 2 to fix this problem. > 2) The new "2018-11-09" REST version introduces support to i) automatically > translate UPNs to OIDs when setting the owner, owning group, or ACL and ii) > optionally translate OIDs to UPNs in the responses when getting the owner, > owning group, or ACL. Configuration will be introduced to optionally > translate OIDs to UPNs in the responses. Since translation has a performance > impact, the default will be to perform no translation and return the OIDs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15872) ABFS: Update to target 2018-11-09 REST version for ADLS Gen 2
[
https://issues.apache.org/jira/browse/HADOOP-15872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694685#comment-16694685
]
Steve Loughran commented on HADOOP-15872:
-
Is the object store ready for this patch now?
If so, this patch is nearly ready. One little checkstyle
{code}
./hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java:92:import
static
org.apache.hadoop.fs.azurebfs.constants.ConfigurationKeys.FS_AZURE_ACCOUNT_AUTH_TYPE_PROPERTY_NAME;:15:
Unused import -
org.apache.hadoop.fs.azurebfs.constants.ConfigurationKeys.FS_AZURE_ACCOUNT_AUTH_TYPE_PROPERTY_NAME.
[UnusedImports]
{code}
> ABFS: Update to target 2018-11-09 REST version for ADLS Gen 2
> -
>
> Key: HADOOP-15872
> URL: https://issues.apache.org/jira/browse/HADOOP-15872
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: Thomas Marquardt
>Assignee: junhua gu
>Priority: Major
> Attachments: HADOOP-15872-001.patch, HADOOP-15872-002.patch,
> HADOOP-15872-003.patch, HADOOP-15872-004.patch
>
>
> This update to the latest REST version (2018-11-09) will make the following
> changes to the ABFS driver:
> 1) The ABFS implementation of getFileStatus currently requires read
> permission. According to HDFS permissions guide, it should only require
> execute on the parent folders (traversal access). A new REST API has been
> introduced in REST version "2018-11-09" of ADLS Gen 2 to fix this problem.
> 2) The new "2018-11-09" REST version introduces support to i) automatically
> translate UPNs to OIDs when setting the owner, owning group, or ACL and ii)
> optionally translate OIDs to UPNs in the responses when getting the owner,
> owning group, or ACL. Configuration will be introduced to optionally
> translate OIDs to UPNs in the responses. Since translation has a performance
> impact, the default will be to perform no translation and return the OIDs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15872) ABFS: Update to target 2018-11-09 REST version for ADLS Gen 2
[
https://issues.apache.org/jira/browse/HADOOP-15872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16688959#comment-16688959
]
Hadoop QA commented on HADOOP-15872:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 42s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 20s{color} | {color:orange} hadoop-tools/hadoop-azure: The patch generated 1
new + 6 unchanged - 0 fixed = 7 total (was 6) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 32s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
17s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 60m 58s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HADOOP-15872 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12948427/HADOOP-15872-004.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 4379a5b2d198 4.4.0-134-generic #160~14.04.1-Ubuntu SMP Fri Aug
17 11:07:07 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / a16aa2f |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15530/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15530/testReport/ |
| Max. process+thread count | 342 (vs. ulimit of 1) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15530/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org
[jira] [Commented] (HADOOP-15872) ABFS: Update to target 2018-11-09 REST version for ADLS Gen 2
[ https://issues.apache.org/jira/browse/HADOOP-15872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16688908#comment-16688908 ] Da Zhou commented on HADOOP-15872: -- Submitting patch 004. - For HNS account using SharedKey, we need to update owner with "$superuser". All tests passed. - When server return owner as "$superuser", we should replace it with local username. *HNS account, using SharedKey* Tests run: 35, Failures: 0, Errors: 0, Skipped: 0 Tests run: 316, Failures: 0, Errors: 0, Skipped: 19 Tests run: 165, Failures: 0, Errors: 0, Skipped: 15 *HNS account, using OAuth* Tests run: 35, Failures: 0, Errors: 0, Skipped: 0 Tests run: 316, Failures: 0, Errors: 0, Skipped: 20 Tests run: 165, Failures: 0, Errors: 0, Skipped: 21 *Non-HNS account, using SharedKey:* Tests run: 35, Failures: 0, Errors: 0, Skipped: 0 Tests run: 316, Failures: 0, Errors: 0, Skipped: 205 Tests run: 165, Failures: 0, Errors: 0, Skipped: 15 > ABFS: Update to target 2018-11-09 REST version for ADLS Gen 2 > - > > Key: HADOOP-15872 > URL: https://issues.apache.org/jira/browse/HADOOP-15872 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Affects Versions: 3.2.0 >Reporter: Thomas Marquardt >Assignee: junhua gu >Priority: Major > Attachments: HADOOP-15872-001.patch, HADOOP-15872-002.patch, > HADOOP-15872-003.patch, HADOOP-15872-004.patch > > > This update to the latest REST version (2018-11-09) will make the following > changes to the ABFS driver: > 1) The ABFS implementation of getFileStatus currently requires read > permission. According to HDFS permissions guide, it should only require > execute on the parent folders (traversal access). A new REST API has been > introduced in REST version "2018-11-09" of ADLS Gen 2 to fix this problem. > 2) The new "2018-11-09" REST version introduces support to i) automatically > translate UPNs to OIDs when setting the owner, owning group, or ACL and ii) > optionally translate OIDs to UPNs in the responses when getting the owner, > owning group, or ACL. Configuration will be introduced to optionally > translate OIDs to UPNs in the responses. Since translation has a performance > impact, the default will be to perform no translation and return the OIDs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15872) ABFS: Update to target 2018-11-09 REST version for ADLS Gen 2
[
https://issues.apache.org/jira/browse/HADOOP-15872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16687202#comment-16687202
]
Da Zhou commented on HADOOP-15872:
--
Hi Steve,
{quote}if you run it now, everything fails with an error about headers : "The
value for one of the HTTP headers is not in the correct format.".
{quote}
Reason: I checked the test tenant you used, it doesn't contain the new service
change related to this patch. I also reran the tests with my test account,
which contains new service change, all tests passed.
{quote}getFileStatus() fails on the cli (hadoop fs -ls; cloudstore store diag)
As far as I can tell, this is a change in the behaviour in the ADLS endpoint
{quote}
Reason: abfs://stevel-testing@ACCOUNT.*blob.core.windows.net*/ was used, this
is for WASB not ABFS. For ABFS you should use *dfs.core.windows.net*
> ABFS: Update to target 2018-11-09 REST version for ADLS Gen 2
> -
>
> Key: HADOOP-15872
> URL: https://issues.apache.org/jira/browse/HADOOP-15872
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: Thomas Marquardt
>Assignee: junhua gu
>Priority: Major
> Attachments: HADOOP-15872-001.patch, HADOOP-15872-002.patch,
> HADOOP-15872-003.patch
>
>
> This update to the latest REST version (2018-11-09) will make the following
> changes to the ABFS driver:
> 1) The ABFS implementation of getFileStatus currently requires read
> permission. According to HDFS permissions guide, it should only require
> execute on the parent folders (traversal access). A new REST API has been
> introduced in REST version "2018-11-09" of ADLS Gen 2 to fix this problem.
> 2) The new "2018-11-09" REST version introduces support to i) automatically
> translate UPNs to OIDs when setting the owner, owning group, or ACL and ii)
> optionally translate OIDs to UPNs in the responses when getting the owner,
> owning group, or ACL. Configuration will be introduced to optionally
> translate OIDs to UPNs in the responses. Since translation has a performance
> impact, the default will be to perform no translation and return the OIDs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15872) ABFS: Update to target 2018-11-09 REST version for ADLS Gen 2
[
https://issues.apache.org/jira/browse/HADOOP-15872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685838#comment-16685838
]
Steve Loughran commented on HADOOP-15872:
-
And, and this is where it gets really interesting, hadoop 3.2 & trunk without
this patch get through all the tests, but getFileStatus() fails on the cli
(hadoop fs -ls; cloudstore store diag)
As far as I can tell, this is a change in the behaviour in the ADLS endpoint
{code:java}
2018-11-13 22:16:15,835 [main] INFO diag.StoreDiag
(DurationInfo.java:(53)) - Starting: Creating filesystem
2018-11-13 22:16:15,936 [main] DEBUG azurebfs.AzureBlobFileSystem
(AzureBlobFileSystem.java:initialize(103)) - Initializing AzureBlobFileSystem
for abfs://stevel-test...@account.blob.core.windows.net/
2018-11-13 22:16:16,161 [main] DEBUG services.AbfsClientThrottlingIntercept
(AbfsClientThrottlingIntercept.java:initializeSingleton(62)) - Client-side
throttling is enabled for the ABFS file system.
2018-11-13 22:16:16,174 [main] INFO diag.StoreDiag
(DurationInfo.java:close(100)) - Creating filesystem: duration 0:00:341
AzureBlobFileSystem{uri=abfs://stevel-test...@account.blob.core.windows.net,
user='stevel', primaryUserGroup='staff'}
2018-11-13 22:16:16,174 [main] INFO diag.StoreDiag
(DurationInfo.java:(53)) - Starting: GetFileStatus
abfs://stevel-test...@account.blob.core.windows.net/
2018-11-13 22:16:16,174 [main] DEBUG azurebfs.AzureBlobFileSystem
(AzureBlobFileSystem.java:getFileStatus(389)) -
AzureBlobFileSystem.getFileStatus path:
abfs://stevel-test...@account.blob.core.windows.net/
2018-11-13 22:16:16,175 [main] DEBUG azurebfs.AzureBlobFileSystem
(AzureBlobFileSystem.java:performAbfsAuthCheck(1025)) - ABFS authorizer is not
initialized. No authorization check will be performed.
2018-11-13 22:16:16,175 [main] DEBUG azurebfs.AzureBlobFileSystemStore
(AzureBlobFileSystemStore.java:getIsNamespaceEnabled(178)) -
getFilesystemProperties for filesystem: stevel-testing
2018-11-13 22:16:16,964 [main] DEBUG services.AbfsClient
(AbfsRestOperation.java:executeHttpOperation(192)) - HttpRequest:
400,,cid=5b89e9e3-b87e-4329-9451-8326331e26b0,rid=3a5caef2-401e-005c-529e-7bb87e00,sent=0,recv=0,HEAD,https://ACCOUNT.blob.core.windows.net/stevel-testing?resource=filesystem=90
2018-11-13 22:16:16,969 [main] INFO diag.StoreDiag
(DurationInfo.java:close(100)) - GetFileStatus
abfs://stevel-test...@account.blob.core.windows.net/: duration 0:00:794
Operation failed: "The value for one of the HTTP headers is not in the correct
format.", 400, HEAD,
https://ACCOUNT.blob.core.windows.net/stevel-testing?resource=filesystem=90
at
org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.execute(AbfsRestOperation.java:134)
at
org.apache.hadoop.fs.azurebfs.services.AbfsClient.getFilesystemProperties(AbfsClient.java:197)
at
org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.getIsNamespaceEnabled(AzureBlobFileSystemStore.java:181)
at
org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.getFileStatus(AzureBlobFileSystemStore.java:454)
at
org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.getFileStatus(AzureBlobFileSystem.java:395)
at
org.apache.hadoop.fs.store.diag.StoreDiag.executeFileSystemOperations(StoreDiag.java:692)
at org.apache.hadoop.fs.store.diag.StoreDiag.run(StoreDiag.java:386)
at org.apache.hadoop.fs.store.diag.StoreDiag.run(StoreDiag.java:331)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
at org.apache.hadoop.fs.store.diag.StoreDiag.exec(StoreDiag.java:989)
at org.apache.hadoop.fs.store.diag.StoreDiag.main(StoreDiag.java:998)
at storediag.main(storediag.java:24)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
2018-11-13 22:16:16,971 [main] INFO util.ExitUtil
(ExitUtil.java:terminate(210)) - Exiting with status -1: Operation failed: "The
value for one of the HTTP headers is not in the correct format.", 400, HEAD,
https://ACCOUNT.blob.core.windows.net/stevel-testing?resource=filesystem=90
2018-11-13 22:16:16,973 [shutdown-hook-0] DEBUG azurebfs.AzureBlobFileSystem
(AzureBlobFileSystem.java:close(383)) - AzureBlobFileSystem.close
{code}
> ABFS: Update to target 2018-11-09 REST version for ADLS Gen 2
> -
>
> Key: HADOOP-15872
> URL: https://issues.apache.org/jira/browse/HADOOP-15872
>
[jira] [Commented] (HADOOP-15872) ABFS: Update to target 2018-11-09 REST version for ADLS Gen 2
[
https://issues.apache.org/jira/browse/HADOOP-15872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685834#comment-16685834
]
Steve Loughran commented on HADOOP-15872:
-
=0.
if you run it now, everything fails with an error about headers : "The value
for one of the HTTP headers is not in the correct format.".
{code}
[ERROR]
testEnsureAclOperationWorksForRoot(org.apache.hadoop.fs.azurebfs.ITestAzureBlobFilesystemAcl)
Time elapsed: 0.032 s <<< ERROR!
Operation failed: "The value for one of the HTTP headers is not in the correct
format.", 400, HEAD,
https://abfsamtest2.dfs.core.windows.net/abfs-testcontainer-dd8b2de2-478b-4f74-8d9b-206a5c888fb4?resource=filesystem=90
at
org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.execute(AbfsRestOperation.java:134)
at
org.apache.hadoop.fs.azurebfs.services.AbfsClient.getFilesystemProperties(AbfsClient.java:198)
at
org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.getFilesystemProperties(AzureBlobFileSystemStore.java:231)
at
org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.fileSystemExists(AzureBlobFileSystem.java:806)
at
org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.initialize(AzureBlobFileSystem.java:114)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3302)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3351)
at org.apache.hadoop.fs.FileSystem$Cache.getUnique(FileSystem.java:3325)
at org.apache.hadoop.fs.FileSystem.newInstance(FileSystem.java:532)
at org.apache.hadoop.fs.FileSystem.newInstance(FileSystem.java:544)
at
org.apache.hadoop.fs.azurebfs.AbstractAbfsIntegrationTest.createFileSystem(AbstractAbfsIntegrationTest.java:215)
at
org.apache.hadoop.fs.azurebfs.AbstractAbfsIntegrationTest.setup(AbstractAbfsIntegrationTest.java:134)
at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:745)
[INFO]
{code}
> ABFS: Update to target 2018-11-09 REST version for ADLS Gen 2
> -
>
> Key: HADOOP-15872
> URL: https://issues.apache.org/jira/browse/HADOOP-15872
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: Thomas Marquardt
>Assignee: junhua gu
>Priority: Major
> Attachments: HADOOP-15872-001.patch, HADOOP-15872-002.patch,
> HADOOP-15872-003.patch
>
>
> This update to the latest REST version (2018-11-09) will make the following
> changes to the ABFS driver:
> 1) The ABFS implementation of getFileStatus currently requires read
> permission. According to HDFS permissions guide, it should only require
> execute on the parent folders (traversal access). A new REST API has been
> introduced in REST version "2018-11-09" of ADLS Gen 2 to fix this problem.
> 2) The new "2018-11-09" REST version introduces support to i) automatically
> translate UPNs to OIDs when setting the owner, owning group, or ACL and ii)
> optionally translate OIDs to UPNs in the responses when getting the owner,
> owning group, or ACL. Configuration will be introduced to optionally
> translate OIDs to UPNs in the responses. Since translation has a performance
> impact, the default will be to perform no translation and return the OIDs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15872) ABFS: Update to target 2018-11-09 REST version for ADLS Gen 2
[ https://issues.apache.org/jira/browse/HADOOP-15872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16683565#comment-16683565 ] Steve Loughran commented on HADOOP-15872: - OK, LGTM, +1 Ping me as soon as the new endpoint is live and we'll switch branch3.2+ to this API > ABFS: Update to target 2018-11-09 REST version for ADLS Gen 2 > - > > Key: HADOOP-15872 > URL: https://issues.apache.org/jira/browse/HADOOP-15872 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Affects Versions: 3.2.0 >Reporter: Thomas Marquardt >Assignee: junhua gu >Priority: Major > Attachments: HADOOP-15872-001.patch, HADOOP-15872-002.patch, > HADOOP-15872-003.patch > > > This update to the latest REST version (2018-11-09) will make the following > changes to the ABFS driver: > 1) The ABFS implementation of getFileStatus currently requires read > permission. According to HDFS permissions guide, it should only require > execute on the parent folders (traversal access). A new REST API has been > introduced in REST version "2018-11-09" of ADLS Gen 2 to fix this problem. > 2) The new "2018-11-09" REST version introduces support to i) automatically > translate UPNs to OIDs when setting the owner, owning group, or ACL and ii) > optionally translate OIDs to UPNs in the responses when getting the owner, > owning group, or ACL. Configuration will be introduced to optionally > translate OIDs to UPNs in the responses. Since translation has a performance > impact, the default will be to perform no translation and return the OIDs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15872) ABFS: Update to target 2018-11-09 REST version for ADLS Gen 2
[
https://issues.apache.org/jira/browse/HADOOP-15872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16682756#comment-16682756
]
Hadoop QA commented on HADOOP-15872:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 44s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 24s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
11s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 53m 21s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HADOOP-15872 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12947728/HADOOP-15872-003.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux fc5741fc1aa5 3.13.0-144-generic #193-Ubuntu SMP Thu Mar 15
17:03:53 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 2664248 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15504/testReport/ |
| Max. process+thread count | 338 (vs. ulimit of 1) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15504/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> ABFS: Update to target 2018-11-09 REST version for ADLS Gen 2
> -
>
> Key: HADOOP-15872
>
[jira] [Commented] (HADOOP-15872) ABFS: Update to target 2018-11-09 REST version for ADLS Gen 2
[ https://issues.apache.org/jira/browse/HADOOP-15872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16682748#comment-16682748 ] Da Zhou commented on HADOOP-15872: -- Updated test committed in HADOOP-15846 to use UUID instead of using literal string "foo". All tests passed: *HNS account, using SharedKey* Tests run: 35, Failures: 0, Errors: 0, Skipped: 0 Tests run: 316, Failures: 0, Errors: 0, Skipped: 19 Tests run: 165, Failures: 0, Errors: 0, Skipped: 15 *HNS account, using OAuth* Tests run: 35, Failures: 0, Errors: 0, Skipped: 0 Tests run: 316, Failures: 0, Errors: 0, Skipped: 20 Tests run: 165, Failures: 0, Errors: 0, Skipped: 21 *Non-HNS account, using SharedKey:* Tests run: 35, Failures: 0, Errors: 0, Skipped: 0 Tests run: 316, Failures: 0, Errors: 0, Skipped: 205 Tests run: 165, Failures: 0, Errors: 0, Skipped: 15 > ABFS: Update to target 2018-11-09 REST version for ADLS Gen 2 > - > > Key: HADOOP-15872 > URL: https://issues.apache.org/jira/browse/HADOOP-15872 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Affects Versions: 3.2.0 >Reporter: Thomas Marquardt >Assignee: junhua gu >Priority: Major > Attachments: HADOOP-15872-001.patch, HADOOP-15872-002.patch, > HADOOP-15872-003.patch > > > This update to the latest REST version (2018-11-09) will make the following > changes to the ABFS driver: > 1) The ABFS implementation of getFileStatus currently requires read > permission. According to HDFS permissions guide, it should only require > execute on the parent folders (traversal access). A new REST API has been > introduced in REST version "2018-11-09" of ADLS Gen 2 to fix this problem. > 2) The new "2018-11-09" REST version introduces support to i) automatically > translate UPNs to OIDs when setting the owner, owning group, or ACL and ii) > optionally translate OIDs to UPNs in the responses when getting the owner, > owning group, or ACL. Configuration will be introduced to optionally > translate OIDs to UPNs in the responses. Since translation has a performance > impact, the default will be to perform no translation and return the OIDs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15872) ABFS: Update to target 2018-11-09 REST version for ADLS Gen 2
[
https://issues.apache.org/jira/browse/HADOOP-15872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681364#comment-16681364
]
Hadoop QA commented on HADOOP-15872:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
31s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 50s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 30s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
11s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 54m 1s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HADOOP-15872 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12947466/HADOOP-15872-002.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 8a80d1dd84a2 3.13.0-144-generic #193-Ubuntu SMP Thu Mar 15
17:03:53 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 9dbb2b6 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15502/testReport/ |
| Max. process+thread count | 339 (vs. ulimit of 1) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15502/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> ABFS: Update to target 2018-11-09 REST version for ADLS Gen 2
> -
>
> Key: HADOOP-15872
>
[jira] [Commented] (HADOOP-15872) ABFS: Update to target 2018-11-09 REST version for ADLS Gen 2
[ https://issues.apache.org/jira/browse/HADOOP-15872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680273#comment-16680273 ] Da Zhou commented on HADOOP-15872: -- Submitting patch 002. - For namespace account, instead of calling API "*GetPathProperties*", it should call "*GetPathStatus*" - I merged new method GetPathStatus to GetPathProperties to avoid duplicate code. All tests passed: XNS account: sharedKey Tests run: 35, Failures: 0, Errors: 0, Skipped: 0 Tests run: 307, Failures: 0, Errors: 0, Skipped: 22 Tests run: 165, Failures: 0, Errors: 0, Skipped: 15 XNS account: OAUTH Tests run: 35, Failures: 0, Errors: 0, Skipped: 0 Tests run: 307, Failures: 0, Errors: 0, Skipped: 23 Tests run: 165, Failures: 0, Errors: 0, Skipped: 21 non-XNS account: SharedKey： Tests run: 35, Failures: 0, Errors: 0, Skipped: 0 Tests run: 307, Failures: 0, Errors: 0, Skipped: 197 Tests run: 165, Failures: 0, Errors: 0, Skipped: 15 > ABFS: Update to target 2018-11-09 REST version for ADLS Gen 2 > - > > Key: HADOOP-15872 > URL: https://issues.apache.org/jira/browse/HADOOP-15872 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Affects Versions: 3.2.0 >Reporter: Thomas Marquardt >Assignee: junhua gu >Priority: Major > Attachments: HADOOP-15872-001.patch, HADOOP-15872-002.patch > > > This update to the latest REST version (2018-11-09) will make the following > changes to the ABFS driver: > 1) The ABFS implementation of getFileStatus currently requires read > permission. According to HDFS permissions guide, it should only require > execute on the parent folders (traversal access). A new REST API has been > introduced in REST version "2018-11-09" of ADLS Gen 2 to fix this problem. > 2) The new "2018-11-09" REST version introduces support to i) automatically > translate UPNs to OIDs when setting the owner, owning group, or ACL and ii) > optionally translate OIDs to UPNs in the responses when getting the owner, > owning group, or ACL. Configuration will be introduced to optionally > translate OIDs to UPNs in the responses. Since translation has a performance > impact, the default will be to perform no translation and return the OIDs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15872) ABFS: Update to target 2018-11-09 REST version for ADLS Gen 2
[ https://issues.apache.org/jira/browse/HADOOP-15872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16663966#comment-16663966 ] Steve Loughran commented on HADOOP-15872: - Noted. Assuming you have your own internal beta-test endpoint: can you confirm that with this patch in, all of the abfs tests completed successfully? > ABFS: Update to target 2018-11-09 REST version for ADLS Gen 2 > - > > Key: HADOOP-15872 > URL: https://issues.apache.org/jira/browse/HADOOP-15872 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Affects Versions: 3.2.0 >Reporter: Thomas Marquardt >Assignee: Thomas Marquardt >Priority: Major > Attachments: HADOOP-15872-001.patch > > > This update to the latest REST version (2018-11-09) will make the following > changes to the ABFS driver: > 1) The ABFS implementation of getFileStatus currently requires read > permission. According to HDFS permissions guide, it should only require > execute on the parent folders (traversal access). A new REST API has been > introduced in REST version "2018-11-09" of ADLS Gen 2 to fix this problem. > 2) The new "2018-11-09" REST version introduces support to i) automatically > translate UPNs to OIDs when setting the owner, owning group, or ACL and ii) > optionally translate OIDs to UPNs in the responses when getting the owner, > owning group, or ACL. Configuration will be introduced to optionally > translate OIDs to UPNs in the responses. Since translation has a performance > impact, the default will be to perform no translation and return the OIDs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
