[jira] [Commented] (HADOOP-14820) Fix for HDFS semantics parity for mkdirs -p
[ https://issues.apache.org/jira/browse/HADOOP-14820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148868#comment-16148868 ] Steve Loughran commented on HADOOP-14820: - Looks pretty good checkstyle: don't worry about the "magic number in tests" one, as it's clearly a false alarm, but the one in src/main is trivial to fix The new test is comparing against /, as far as I can see. Looking at this & the existing checks, I don't see any which try to play with permissions and directories partway up the tree other than / {code} Path path1 = "/path1" fs.mkdir(path1) make-read-only(path1) // immediate child to fail LambdaTestUtils.intercept(WasbAuthorizationException.class, () -> fs.mkdir(new Path(path1, "/path2")); // implement as a Callable for Java 7 // grandchild to fail LambdaTestUtils.intercept(WasbAuthorizationException.class, () -> fs.mkdir(new Path(path1, "/path2/path3")); {code} > Fix for HDFS semantics parity for mkdirs -p > > > Key: HADOOP-14820 > URL: https://issues.apache.org/jira/browse/HADOOP-14820 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: Sivaguru Sankaridurg >Assignee: Sivaguru Sankaridurg > Labels: azure, fs, secure, wasb > Attachments: HADOOP-14820.001.patch, HADOOP-14820.002.patch, > HADOOP-14820.003.patch > > > No authorization checks should be made when a user tries to create (mkdirs > -p) an existing folder hierarchy. > For example, if we start with _/home/hdiuser/prefix_ pre-created, and do the > following operations, the results should be as shown below. > {noformat} > hdiuser@hn0-0d2f67:~$ sudo chown root:root prefix > hdiuser@hn0-0d2f67:~$ sudo chmod 555 prefix > hdiuser@hn0-0d2f67:~$ ls -l > dr-xr-xr-x 3 rootroot 4096 Aug 29 08:25 prefix > hdiuser@hn0-0d2f67:~$ mkdir -p /home > hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser > hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser/prefix > hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser/prefix/1 > mkdir: cannot create directory â/home/hdiuser/prefix/1â: Permission denied > The first three mkdirs succeed, because the ancestor is already present. The > fourth one fails because of a permission check against the (shorter) ancestor > (as compared to the path being created). > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14820) Fix for HDFS semantics parity for mkdirs -p
[ https://issues.apache.org/jira/browse/HADOOP-14820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148536#comment-16148536 ] Hadoop QA commented on HADOOP-14820: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m 43s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 20s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 15s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 21s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 28s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 18s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 16s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 12s{color} | {color:orange} hadoop-tools/hadoop-azure: The patch generated 2 new + 80 unchanged - 1 fixed = 82 total (was 81) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 18s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 33s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 2s{color} | {color:green} hadoop-azure in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 20m 55s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:71bbb86 | | JIRA Issue | HADOOP-14820 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12884602/HADOOP-14820.003.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 622592cd1c26 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 71bbb86 | | Default Java | 1.8.0_144 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/PreCommit-HADOOP-Build/13143/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-azure.txt | | Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/13143/testReport/ | | modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure | | Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/13143/console | | Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > Fix for HDFS semantics parity for mkdirs -p > > > Key: HADOOP-14820 > URL: https://issues.apache.org/jira/browse/HADOOP-14820 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: Sivaguru Sankaridurg >Assignee: Sivaguru Sankaridurg > Labels: azure, fs, secure, wasb > Attachments:
[jira] [Commented] (HADOOP-14820) Fix for HDFS semantics parity for mkdirs -p
[ https://issues.apache.org/jira/browse/HADOOP-14820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148486#comment-16148486 ] Sivaguru Sankaridurg commented on HADOOP-14820: --- *NativeAzureFileSystem.java L1714 - * Thanks for pointing this out. The fix to {noformat}getAncestor(){noformat} should take care of this. This is the correct fix. The call to {noformat}performAuthCheck{noformat} on L1714 is required. *L2426* - This has been fixed. Please see changes to {noformat}getAncestor{noformat} in 003.patch. The naming is consistent with the definition mentioned at [https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.html#Permission_Checks]. {noformat} Ancestor: The last existing component of the requested path. For example, for the path /foo/bar/baz, the ancestor path is /foo/bar if /foo/bar exists. The ancestor path is /foo if /foo exists but /foo/bar does not exist.{noformat} *L2459* - Same as above. > Fix for HDFS semantics parity for mkdirs -p > > > Key: HADOOP-14820 > URL: https://issues.apache.org/jira/browse/HADOOP-14820 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: Sivaguru Sankaridurg >Assignee: Sivaguru Sankaridurg > Labels: azure, fs, secure, wasb > Attachments: HADOOP-14820.001.patch, HADOOP-14820.002.patch > > > No authorization checks should be made when a user tries to create (mkdirs > -p) an existing folder hierarchy. > For example, if we start with _/home/hdiuser/prefix_ pre-created, and do the > following operations, the results should be as shown below. > {noformat} > hdiuser@hn0-0d2f67:~$ sudo chown root:root prefix > hdiuser@hn0-0d2f67:~$ sudo chmod 555 prefix > hdiuser@hn0-0d2f67:~$ ls -l > dr-xr-xr-x 3 rootroot 4096 Aug 29 08:25 prefix > hdiuser@hn0-0d2f67:~$ mkdir -p /home > hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser > hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser/prefix > hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser/prefix/1 > mkdir: cannot create directory â/home/hdiuser/prefix/1â: Permission denied > The first three mkdirs succeed, because the ancestor is already present. The > fourth one fails because of a permission check against the (shorter) ancestor > (as compared to the path being created). > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14820) Fix for HDFS semantics parity for mkdirs -p
[ https://issues.apache.org/jira/browse/HADOOP-14820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16147860#comment-16147860 ] Thomas Marquardt commented on HADOOP-14820: --- Looks good. I have the following feedback: *NativeAzureFileSystem.java* *L1714* - When the file already exists and overwrite is true, both {{performAuthCheck}} calls (L1700 and L1714) are checking permissions on the file. The {{performAuthCheck}} on L1714 can be removed. *L2426* - The method {{getAncestor}} no longer returns the ancestor or parent path; instead, it returns the first path segment that exists. I recommend renaming it {{getFirstPathSegmentThatExists}}. *L2459* - The {{ancestor}} field is actual the first path segment that exists, which may be the file itself. method {{getAncestor}} no longer returns the ancestor or parent path; instead, it returns the first path segment that exists. I recommend renaming it {{firstExistingPathSegment}}. . > Fix for HDFS semantics parity for mkdirs -p > > > Key: HADOOP-14820 > URL: https://issues.apache.org/jira/browse/HADOOP-14820 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: Sivaguru Sankaridurg >Assignee: Sivaguru Sankaridurg > Labels: azure, fs, secure, wasb > Attachments: HADOOP-14820.001.patch, HADOOP-14820.002.patch > > > No authorization checks should be made when a user tries to create (mkdirs > -p) an existing folder hierarchy. > For example, if we start with _/home/hdiuser/prefix_ pre-created, and do the > following operations, the results should be as shown below. > {noformat} > hdiuser@hn0-0d2f67:~$ sudo chown root:root prefix > hdiuser@hn0-0d2f67:~$ sudo chmod 555 prefix > hdiuser@hn0-0d2f67:~$ ls -l > dr-xr-xr-x 3 rootroot 4096 Aug 29 08:25 prefix > hdiuser@hn0-0d2f67:~$ mkdir -p /home > hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser > hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser/prefix > hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser/prefix/1 > mkdir: cannot create directory â/home/hdiuser/prefix/1â: Permission denied > The first three mkdirs succeed, because the ancestor is already present. The > fourth one fails because of a permission check against the (shorter) ancestor > (as compared to the path being created). > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14820) Fix for HDFS semantics parity for mkdirs -p
[ https://issues.apache.org/jira/browse/HADOOP-14820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16147537#comment-16147537 ] Hadoop QA commented on HADOOP-14820: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m 15s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 28s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 18s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 29s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 41s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 25s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 24s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 24s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 16s{color} | {color:orange} hadoop-tools/hadoop-azure: The patch generated 2 new + 80 unchanged - 1 fixed = 82 total (was 81) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 28s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 46s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 29s{color} | {color:green} hadoop-azure in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 19s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 27m 39s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | HADOOP-14820 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12884498/HADOOP-14820.002.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 47dc7afe7f30 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 9992675 | | Default Java | 1.8.0_144 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/PreCommit-HADOOP-Build/13134/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-azure.txt | | Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/13134/testReport/ | | modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure | | Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/13134/console | | Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > Fix for HDFS semantics parity for mkdirs -p > > > Key: HADOOP-14820 > URL: https://issues.apache.org/jira/browse/HADOOP-14820 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: Sivaguru Sankaridurg >Assignee: Sivaguru Sankaridurg > Labels: azure, fs, secure, wasb > Attachments:
[jira] [Commented] (HADOOP-14820) Fix for HDFS semantics parity for mkdirs -p
[ https://issues.apache.org/jira/browse/HADOOP-14820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16147233#comment-16147233 ] Hadoop QA commented on HADOOP-14820: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m 59s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 19s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 14s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 21s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 28s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 17s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 16s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 11s{color} | {color:orange} hadoop-tools/hadoop-azure: The patch generated 2 new + 80 unchanged - 1 fixed = 82 total (was 81) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 18s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 33s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 1s{color} | {color:green} hadoop-azure in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 14s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 21m 9s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | HADOOP-14820 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12884469/HADOOP-14820.001.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 3eb9cac08840 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 200b113 | | Default Java | 1.8.0_144 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/PreCommit-HADOOP-Build/13132/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-azure.txt | | Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/13132/testReport/ | | modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure | | Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/13132/console | | Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > Fix for HDFS semantics parity for mkdirs -p > > > Key: HADOOP-14820 > URL: https://issues.apache.org/jira/browse/HADOOP-14820 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: Sivaguru Sankaridurg >Assignee: Sivaguru Sankaridurg > Labels: azure, fs, secure, wasb > Attachments: