[jira] [Commented] (HDDS-2114) Rename does not preserve non-explicitly created interim directories
[ https://issues.apache.org/jira/browse/HDDS-2114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932245#comment-16932245 ] Steve Loughran commented on HDDS-2114: -- I see you found the bit in the S3A FS which does it. Yes, we recreate the markers. There's a cost in time and $, but a fair amount of people and code expect those parent directories to still be there. Pity > Rename does not preserve non-explicitly created interim directories > --- > > Key: HDDS-2114 > URL: https://issues.apache.org/jira/browse/HDDS-2114 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Istvan Fajth >Assignee: Lokesh Jain >Priority: Critical > Labels: pull-request-available > Fix For: 0.4.1, 0.5.0 > > Attachments: demonstrative_test.patch > > Time Spent: 1.5h > Remaining Estimate: 0h > > I am attaching a patch that adds a test that demonstrates the problem. > The scenario is coming from the way how Hive implements acid transactions > with the ORC table format, but the test is redacted to the simplest possible > code that reproduces the issue. > The scenario: > * Given a 3 level directory structure, where the top level directory was > explicitly created, and the interim directory is implicitly created (for > example either by creating a file with create("/top/interim/file") or by > creating a directory with mkdirs("top/interim/dir")) > * When the leaf is moved out from the implicitly created directory making > this directory an empty directory > * Then a FileNotFoundException is thrown when getFileStatus or listStatus is > called on the interim directory. > The expected behaviour: > after the directory is becoming empty, the directory should still be part of > the file system, moreover an empty FileStatus array should be returned when > listStatus is called on it, and also a valid FileStatus object should be > returned when getFileStatus is called on it. > > > As this issue is present with Hive, and as this is how a FileSystem is > expected to work this seems to be an at least critical issue as I see, please > feel free to change the priority if needed. > Also please note that, if the interim directory is explicitly created with > mkdirs("top/interim") before creating the leaf, then the issue does not > appear. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2114) Rename does not preserve non-explicitly created interim directories
[ https://issues.apache.org/jira/browse/HDDS-2114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931246#comment-16931246 ] Istvan Fajth commented on HDDS-2114: I have tested the fix on a real cluster with Hive and the workflow that have failed and got us to this issue is working properly after the fix was applied. Thank you [~ljain] for working on it, thanks [~msingh] for the review and commit! > Rename does not preserve non-explicitly created interim directories > --- > > Key: HDDS-2114 > URL: https://issues.apache.org/jira/browse/HDDS-2114 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Istvan Fajth >Assignee: Lokesh Jain >Priority: Critical > Labels: pull-request-available > Fix For: 0.5.0 > > Attachments: demonstrative_test.patch > > Time Spent: 1.5h > Remaining Estimate: 0h > > I am attaching a patch that adds a test that demonstrates the problem. > The scenario is coming from the way how Hive implements acid transactions > with the ORC table format, but the test is redacted to the simplest possible > code that reproduces the issue. > The scenario: > * Given a 3 level directory structure, where the top level directory was > explicitly created, and the interim directory is implicitly created (for > example either by creating a file with create("/top/interim/file") or by > creating a directory with mkdirs("top/interim/dir")) > * When the leaf is moved out from the implicitly created directory making > this directory an empty directory > * Then a FileNotFoundException is thrown when getFileStatus or listStatus is > called on the interim directory. > The expected behaviour: > after the directory is becoming empty, the directory should still be part of > the file system, moreover an empty FileStatus array should be returned when > listStatus is called on it, and also a valid FileStatus object should be > returned when getFileStatus is called on it. > > > As this issue is present with Hive, and as this is how a FileSystem is > expected to work this seems to be an at least critical issue as I see, please > feel free to change the priority if needed. > Also please note that, if the interim directory is explicitly created with > mkdirs("top/interim") before creating the leaf, then the issue does not > appear. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2114) Rename does not preserve non-explicitly created interim directories
[ https://issues.apache.org/jira/browse/HDDS-2114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931220#comment-16931220 ] Hudson commented on HDDS-2114: -- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #17311 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17311/]) HDDS-2114. Rename does not preserve non-explicitly created interim (msingh: rev 292bce7908bf4830c793a3f4e80376819c038379) * (edit) hadoop-ozone/ozonefs/src/main/java/org/apache/hadoop/fs/ozone/BasicOzoneFileSystem.java * (edit) hadoop-ozone/ozonefs/src/test/java/org/apache/hadoop/fs/ozone/TestOzoneFileSystem.java > Rename does not preserve non-explicitly created interim directories > --- > > Key: HDDS-2114 > URL: https://issues.apache.org/jira/browse/HDDS-2114 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Istvan Fajth >Assignee: Lokesh Jain >Priority: Critical > Labels: pull-request-available > Fix For: 0.5.0 > > Attachments: demonstrative_test.patch > > Time Spent: 1.5h > Remaining Estimate: 0h > > I am attaching a patch that adds a test that demonstrates the problem. > The scenario is coming from the way how Hive implements acid transactions > with the ORC table format, but the test is redacted to the simplest possible > code that reproduces the issue. > The scenario: > * Given a 3 level directory structure, where the top level directory was > explicitly created, and the interim directory is implicitly created (for > example either by creating a file with create("/top/interim/file") or by > creating a directory with mkdirs("top/interim/dir")) > * When the leaf is moved out from the implicitly created directory making > this directory an empty directory > * Then a FileNotFoundException is thrown when getFileStatus or listStatus is > called on the interim directory. > The expected behaviour: > after the directory is becoming empty, the directory should still be part of > the file system, moreover an empty FileStatus array should be returned when > listStatus is called on it, and also a valid FileStatus object should be > returned when getFileStatus is called on it. > > > As this issue is present with Hive, and as this is how a FileSystem is > expected to work this seems to be an at least critical issue as I see, please > feel free to change the priority if needed. > Also please note that, if the interim directory is explicitly created with > mkdirs("top/interim") before creating the leaf, then the issue does not > appear. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2114) Rename does not preserve non-explicitly created interim directories
[ https://issues.apache.org/jira/browse/HDDS-2114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931105#comment-16931105 ] Mukul Kumar Singh commented on HDDS-2114: - The build works out locally and hence committed this to trunk > Rename does not preserve non-explicitly created interim directories > --- > > Key: HDDS-2114 > URL: https://issues.apache.org/jira/browse/HDDS-2114 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Istvan Fajth >Assignee: Lokesh Jain >Priority: Critical > Labels: pull-request-available > Fix For: 0.5.0 > > Attachments: demonstrative_test.patch > > Time Spent: 1h 10m > Remaining Estimate: 0h > > I am attaching a patch that adds a test that demonstrates the problem. > The scenario is coming from the way how Hive implements acid transactions > with the ORC table format, but the test is redacted to the simplest possible > code that reproduces the issue. > The scenario: > * Given a 3 level directory structure, where the top level directory was > explicitly created, and the interim directory is implicitly created (for > example either by creating a file with create("/top/interim/file") or by > creating a directory with mkdirs("top/interim/dir")) > * When the leaf is moved out from the implicitly created directory making > this directory an empty directory > * Then a FileNotFoundException is thrown when getFileStatus or listStatus is > called on the interim directory. > The expected behaviour: > after the directory is becoming empty, the directory should still be part of > the file system, moreover an empty FileStatus array should be returned when > listStatus is called on it, and also a valid FileStatus object should be > returned when getFileStatus is called on it. > > > As this issue is present with Hive, and as this is how a FileSystem is > expected to work this seems to be an at least critical issue as I see, please > feel free to change the priority if needed. > Also please note that, if the interim directory is explicitly created with > mkdirs("top/interim") before creating the leaf, then the issue does not > appear. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2114) Rename does not preserve non-explicitly created interim directories
[ https://issues.apache.org/jira/browse/HDDS-2114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16929287#comment-16929287 ] Hadoop QA commented on HDDS-2114: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 19m 7s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 0s{color} | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 30s{color} | {color:red} hadoop-ozone in trunk failed. {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 19s{color} | {color:red} hadoop-ozone in trunk failed. {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 35s{color} | {color:orange} The patch fails to run checkstyle in hadoop-ozone {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 0s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 15m 59s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 14s{color} | {color:red} hadoop-hdds in trunk failed. {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 14s{color} | {color:red} hadoop-ozone in trunk failed. {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 3m 4s{color} | {color:blue} Used deprecated FindBugs config; considering switching to SpotBugs. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 23s{color} | {color:red} hadoop-ozone in trunk failed. {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 31s{color} | {color:red} hadoop-ozone in the patch failed. {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 21s{color} | {color:red} hadoop-ozone in the patch failed. {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 21s{color} | {color:red} hadoop-ozone in the patch failed. {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 26s{color} | {color:orange} The patch fails to run checkstyle in hadoop-ozone {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 0s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 40s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 14s{color} | {color:red} hadoop-hdds in the patch failed. {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 14s{color} | {color:red} hadoop-ozone in the patch failed. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 23s{color} | {color:red} hadoop-ozone in the patch failed. {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 37s{color} | {color:red} hadoop-hdds in the patch failed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 25s{color} | {color:red} hadoop-ozone in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 30s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 68m 28s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.ozone.container.keyvalue.TestKeyValueContainer | | | hadoop.ozone.container.ozoneimpl.TestOzoneContainer | \\ \\ || Subsystem || Report/Notes || | Docker | Client=19.03.2 Server=19.03.2 base: https://builds.apache.org/job/PreCommit-HDDS-Build/2779/artifact/out/Dockerfile | | JIRA Issue | HDDS-2114 | | JIRA Patch URL |
[jira] [Commented] (HDDS-2114) Rename does not preserve non-explicitly created interim directories
[ https://issues.apache.org/jira/browse/HDDS-2114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16929269#comment-16929269 ] Hadoop QA commented on HDDS-2114: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 36s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 0s{color} | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 31s{color} | {color:red} hadoop-ozone in trunk failed. {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 19s{color} | {color:red} hadoop-ozone in trunk failed. {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 4s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 0s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 14m 56s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 13s{color} | {color:red} hadoop-hdds in trunk failed. {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 13s{color} | {color:red} hadoop-ozone in trunk failed. {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 2m 39s{color} | {color:blue} Used deprecated FindBugs config; considering switching to SpotBugs. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 22s{color} | {color:red} hadoop-ozone in trunk failed. {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 30s{color} | {color:red} hadoop-ozone in the patch failed. {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 20s{color} | {color:red} hadoop-ozone in the patch failed. {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 20s{color} | {color:red} hadoop-ozone in the patch failed. {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 26s{color} | {color:orange} hadoop-ozone: The patch generated 2 new + 121 unchanged - 0 fixed = 123 total (was 121) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 0s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 56s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 13s{color} | {color:red} hadoop-hdds in the patch failed. {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 13s{color} | {color:red} hadoop-ozone in the patch failed. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 21s{color} | {color:red} hadoop-ozone in the patch failed. {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 17s{color} | {color:red} hadoop-hdds in the patch failed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 24s{color} | {color:red} hadoop-ozone in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 28s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 47m 40s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.ozone.container.keyvalue.TestKeyValueContainer | | | hadoop.ozone.container.ozoneimpl.TestOzoneContainer | \\ \\ || Subsystem || Report/Notes || | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1440/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1440 | | JIRA Issue | HDDS-2114 | | Optional Tests |