[jira] [Work logged] (HDDS-2114) Rename does not preserve non-explicitly created interim directories
[
https://issues.apache.org/jira/browse/HDDS-2114?focusedWorklogId=313524&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-313524
]
ASF GitHub Bot logged work on HDDS-2114:
Author: ASF GitHub Bot
Created on: 17/Sep/19 06:54
Start Date: 17/Sep/19 06:54
Worklog Time Spent: 10m
Work Description: lokeshj1703 commented on pull request #1440: HDDS-2114:
Rename does not preserve non-explicitly created interim directories
URL: https://github.com/apache/hadoop/pull/1440
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 313524)
Time Spent: 1.5h (was: 1h 20m)
> Rename does not preserve non-explicitly created interim directories
> ---
>
> Key: HDDS-2114
> URL: https://issues.apache.org/jira/browse/HDDS-2114
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Istvan Fajth
>Assignee: Lokesh Jain
>Priority: Critical
> Labels: pull-request-available
> Fix For: 0.5.0
>
> Attachments: demonstrative_test.patch
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> I am attaching a patch that adds a test that demonstrates the problem.
> The scenario is coming from the way how Hive implements acid transactions
> with the ORC table format, but the test is redacted to the simplest possible
> code that reproduces the issue.
> The scenario:
> * Given a 3 level directory structure, where the top level directory was
> explicitly created, and the interim directory is implicitly created (for
> example either by creating a file with create("/top/interim/file") or by
> creating a directory with mkdirs("top/interim/dir"))
> * When the leaf is moved out from the implicitly created directory making
> this directory an empty directory
> * Then a FileNotFoundException is thrown when getFileStatus or listStatus is
> called on the interim directory.
> The expected behaviour:
> after the directory is becoming empty, the directory should still be part of
> the file system, moreover an empty FileStatus array should be returned when
> listStatus is called on it, and also a valid FileStatus object should be
> returned when getFileStatus is called on it.
>
>
> As this issue is present with Hive, and as this is how a FileSystem is
> expected to work this seems to be an at least critical issue as I see, please
> feel free to change the priority if needed.
> Also please note that, if the interim directory is explicitly created with
> mkdirs("top/interim") before creating the leaf, then the issue does not
> appear.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-2114) Rename does not preserve non-explicitly created interim directories
[
https://issues.apache.org/jira/browse/HDDS-2114?focusedWorklogId=313522&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-313522
]
ASF GitHub Bot logged work on HDDS-2114:
Author: ASF GitHub Bot
Created on: 17/Sep/19 06:51
Start Date: 17/Sep/19 06:51
Worklog Time Spent: 10m
Work Description: lokeshj1703 commented on issue #1440: HDDS-2114: Rename
does not preserve non-explicitly created interim directories
URL: https://github.com/apache/hadoop/pull/1440#issuecomment-532086334
@anuengineer Yes, its the same problem as you described. After rename if the
sources's parent has no remaining children, then S3A does a mkdir for the
parent. The PR does exactly the same for ozonefs.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 313522)
Time Spent: 1h 20m (was: 1h 10m)
> Rename does not preserve non-explicitly created interim directories
> ---
>
> Key: HDDS-2114
> URL: https://issues.apache.org/jira/browse/HDDS-2114
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Istvan Fajth
>Assignee: Lokesh Jain
>Priority: Critical
> Labels: pull-request-available
> Fix For: 0.5.0
>
> Attachments: demonstrative_test.patch
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> I am attaching a patch that adds a test that demonstrates the problem.
> The scenario is coming from the way how Hive implements acid transactions
> with the ORC table format, but the test is redacted to the simplest possible
> code that reproduces the issue.
> The scenario:
> * Given a 3 level directory structure, where the top level directory was
> explicitly created, and the interim directory is implicitly created (for
> example either by creating a file with create("/top/interim/file") or by
> creating a directory with mkdirs("top/interim/dir"))
> * When the leaf is moved out from the implicitly created directory making
> this directory an empty directory
> * Then a FileNotFoundException is thrown when getFileStatus or listStatus is
> called on the interim directory.
> The expected behaviour:
> after the directory is becoming empty, the directory should still be part of
> the file system, moreover an empty FileStatus array should be returned when
> listStatus is called on it, and also a valid FileStatus object should be
> returned when getFileStatus is called on it.
>
>
> As this issue is present with Hive, and as this is how a FileSystem is
> expected to work this seems to be an at least critical issue as I see, please
> feel free to change the priority if needed.
> Also please note that, if the interim directory is explicitly created with
> mkdirs("top/interim") before creating the leaf, then the issue does not
> appear.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-2114) Rename does not preserve non-explicitly created interim directories
[
https://issues.apache.org/jira/browse/HDDS-2114?focusedWorklogId=313338&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-313338
]
ASF GitHub Bot logged work on HDDS-2114:
Author: ASF GitHub Bot
Created on: 16/Sep/19 22:37
Start Date: 16/Sep/19 22:37
Worklog Time Spent: 10m
Work Description: anuengineer commented on issue #1440: HDDS-2114: Rename
does not preserve non-explicitly created interim directories
URL: https://github.com/apache/hadoop/pull/1440#issuecomment-531984790
I am going to +1 this. Since we want to make sure Hive works.
I just want to understand this more clearly. The issue is really that if we
were a real file system, then there is nothing called an implicit path. Since
we are an object store, there is a notion of a implicitly created file path (in
this case the intermediary directories). I am guessing that S3AFS has the same
problem, and either Hive has a workaround for this, or S3A is doing something
clever. Do we know how Hive works on S3?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 313338)
Time Spent: 1h 10m (was: 1h)
> Rename does not preserve non-explicitly created interim directories
> ---
>
> Key: HDDS-2114
> URL: https://issues.apache.org/jira/browse/HDDS-2114
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Istvan Fajth
>Assignee: Lokesh Jain
>Priority: Critical
> Labels: pull-request-available
> Attachments: demonstrative_test.patch
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> I am attaching a patch that adds a test that demonstrates the problem.
> The scenario is coming from the way how Hive implements acid transactions
> with the ORC table format, but the test is redacted to the simplest possible
> code that reproduces the issue.
> The scenario:
> * Given a 3 level directory structure, where the top level directory was
> explicitly created, and the interim directory is implicitly created (for
> example either by creating a file with create("/top/interim/file") or by
> creating a directory with mkdirs("top/interim/dir"))
> * When the leaf is moved out from the implicitly created directory making
> this directory an empty directory
> * Then a FileNotFoundException is thrown when getFileStatus or listStatus is
> called on the interim directory.
> The expected behaviour:
> after the directory is becoming empty, the directory should still be part of
> the file system, moreover an empty FileStatus array should be returned when
> listStatus is called on it, and also a valid FileStatus object should be
> returned when getFileStatus is called on it.
>
>
> As this issue is present with Hive, and as this is how a FileSystem is
> expected to work this seems to be an at least critical issue as I see, please
> feel free to change the priority if needed.
> Also please note that, if the interim directory is explicitly created with
> mkdirs("top/interim") before creating the leaf, then the issue does not
> appear.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-2114) Rename does not preserve non-explicitly created interim directories
[
https://issues.apache.org/jira/browse/HDDS-2114?focusedWorklogId=313337&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-313337
]
ASF GitHub Bot logged work on HDDS-2114:
Author: ASF GitHub Bot
Created on: 16/Sep/19 22:36
Start Date: 16/Sep/19 22:36
Worklog Time Spent: 10m
Work Description: anuengineer commented on issue #1440: HDDS-2114: Rename
does not preserve non-explicitly created interim directories
URL: https://github.com/apache/hadoop/pull/1440#issuecomment-531984790
I am going to +1 this. Since we want to make sure Hive works.
I just want to understand this more clearly. The issue is really that if we
were a real file system, then there is nothing called an implicit path. Since
we are an object store, there is a notion of a implicitly created file system.
I am guess that S3AFS has the same problem, and either Hive has a workaround
for this, or S3A is doing something really clever. Do we know how Hive works on
S3?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 313337)
Time Spent: 1h (was: 50m)
> Rename does not preserve non-explicitly created interim directories
> ---
>
> Key: HDDS-2114
> URL: https://issues.apache.org/jira/browse/HDDS-2114
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Istvan Fajth
>Assignee: Lokesh Jain
>Priority: Critical
> Labels: pull-request-available
> Attachments: demonstrative_test.patch
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> I am attaching a patch that adds a test that demonstrates the problem.
> The scenario is coming from the way how Hive implements acid transactions
> with the ORC table format, but the test is redacted to the simplest possible
> code that reproduces the issue.
> The scenario:
> * Given a 3 level directory structure, where the top level directory was
> explicitly created, and the interim directory is implicitly created (for
> example either by creating a file with create("/top/interim/file") or by
> creating a directory with mkdirs("top/interim/dir"))
> * When the leaf is moved out from the implicitly created directory making
> this directory an empty directory
> * Then a FileNotFoundException is thrown when getFileStatus or listStatus is
> called on the interim directory.
> The expected behaviour:
> after the directory is becoming empty, the directory should still be part of
> the file system, moreover an empty FileStatus array should be returned when
> listStatus is called on it, and also a valid FileStatus object should be
> returned when getFileStatus is called on it.
>
>
> As this issue is present with Hive, and as this is how a FileSystem is
> expected to work this seems to be an at least critical issue as I see, please
> feel free to change the priority if needed.
> Also please note that, if the interim directory is explicitly created with
> mkdirs("top/interim") before creating the leaf, then the issue does not
> appear.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-2114) Rename does not preserve non-explicitly created interim directories
[
https://issues.apache.org/jira/browse/HDDS-2114?focusedWorklogId=313033&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-313033
]
ASF GitHub Bot logged work on HDDS-2114:
Author: ASF GitHub Bot
Created on: 16/Sep/19 14:59
Start Date: 16/Sep/19 14:59
Worklog Time Spent: 10m
Work Description: elek commented on issue #1440: HDDS-2114: Rename does
not preserve non-explicitly created interim directories
URL: https://github.com/apache/hadoop/pull/1440#issuecomment-531816460
/retest
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 313033)
Time Spent: 50m (was: 40m)
> Rename does not preserve non-explicitly created interim directories
> ---
>
> Key: HDDS-2114
> URL: https://issues.apache.org/jira/browse/HDDS-2114
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Istvan Fajth
>Assignee: Lokesh Jain
>Priority: Critical
> Labels: pull-request-available
> Attachments: demonstrative_test.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> I am attaching a patch that adds a test that demonstrates the problem.
> The scenario is coming from the way how Hive implements acid transactions
> with the ORC table format, but the test is redacted to the simplest possible
> code that reproduces the issue.
> The scenario:
> * Given a 3 level directory structure, where the top level directory was
> explicitly created, and the interim directory is implicitly created (for
> example either by creating a file with create("/top/interim/file") or by
> creating a directory with mkdirs("top/interim/dir"))
> * When the leaf is moved out from the implicitly created directory making
> this directory an empty directory
> * Then a FileNotFoundException is thrown when getFileStatus or listStatus is
> called on the interim directory.
> The expected behaviour:
> after the directory is becoming empty, the directory should still be part of
> the file system, moreover an empty FileStatus array should be returned when
> listStatus is called on it, and also a valid FileStatus object should be
> returned when getFileStatus is called on it.
>
>
> As this issue is present with Hive, and as this is how a FileSystem is
> expected to work this seems to be an at least critical issue as I see, please
> feel free to change the priority if needed.
> Also please note that, if the interim directory is explicitly created with
> mkdirs("top/interim") before creating the leaf, then the issue does not
> appear.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-2114) Rename does not preserve non-explicitly created interim directories
[
https://issues.apache.org/jira/browse/HDDS-2114?focusedWorklogId=312424&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-312424
]
ASF GitHub Bot logged work on HDDS-2114:
Author: ASF GitHub Bot
Created on: 13/Sep/19 22:58
Start Date: 13/Sep/19 22:58
Worklog Time Spent: 10m
Work Description: xiaoyuyao commented on pull request #1440: HDDS-2114:
Rename does not preserve non-explicitly created interim directories
URL: https://github.com/apache/hadoop/pull/1440#discussion_r324392898
##
File path:
hadoop-ozone/ozonefs/src/main/java/org/apache/hadoop/fs/ozone/BasicOzoneFileSystem.java
##
@@ -374,7 +374,11 @@ public boolean rename(Path src, Path dst) throws
IOException {
}
}
RenameIterator iterator = new RenameIterator(src, dst);
-return iterator.iterate();
+boolean result = iterator.iterate();
+if (result) {
+ createFakeParentDirectory(src);
Review comment:
should we createFakeParentDirectory for dst as it is the rename result?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 312424)
Time Spent: 40m (was: 0.5h)
> Rename does not preserve non-explicitly created interim directories
> ---
>
> Key: HDDS-2114
> URL: https://issues.apache.org/jira/browse/HDDS-2114
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Istvan Fajth
>Assignee: Lokesh Jain
>Priority: Critical
> Labels: pull-request-available
> Attachments: demonstrative_test.patch
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> I am attaching a patch that adds a test that demonstrates the problem.
> The scenario is coming from the way how Hive implements acid transactions
> with the ORC table format, but the test is redacted to the simplest possible
> code that reproduces the issue.
> The scenario:
> * Given a 3 level directory structure, where the top level directory was
> explicitly created, and the interim directory is implicitly created (for
> example either by creating a file with create("/top/interim/file") or by
> creating a directory with mkdirs("top/interim/dir"))
> * When the leaf is moved out from the implicitly created directory making
> this directory an empty directory
> * Then a FileNotFoundException is thrown when getFileStatus or listStatus is
> called on the interim directory.
> The expected behaviour:
> after the directory is becoming empty, the directory should still be part of
> the file system, moreover an empty FileStatus array should be returned when
> listStatus is called on it, and also a valid FileStatus object should be
> returned when getFileStatus is called on it.
>
>
> As this issue is present with Hive, and as this is how a FileSystem is
> expected to work this seems to be an at least critical issue as I see, please
> feel free to change the priority if needed.
> Also please note that, if the interim directory is explicitly created with
> mkdirs("top/interim") before creating the leaf, then the issue does not
> appear.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-2114) Rename does not preserve non-explicitly created interim directories
[
https://issues.apache.org/jira/browse/HDDS-2114?focusedWorklogId=312420&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-312420
]
ASF GitHub Bot logged work on HDDS-2114:
Author: ASF GitHub Bot
Created on: 13/Sep/19 22:55
Start Date: 13/Sep/19 22:55
Worklog Time Spent: 10m
Work Description: xiaoyuyao commented on pull request #1440: HDDS-2114:
Rename does not preserve non-explicitly created interim directories
URL: https://github.com/apache/hadoop/pull/1440#discussion_r324392898
##
File path:
hadoop-ozone/ozonefs/src/main/java/org/apache/hadoop/fs/ozone/BasicOzoneFileSystem.java
##
@@ -374,7 +374,11 @@ public boolean rename(Path src, Path dst) throws
IOException {
}
}
RenameIterator iterator = new RenameIterator(src, dst);
-return iterator.iterate();
+boolean result = iterator.iterate();
+if (result) {
+ createFakeParentDirectory(src);
Review comment:
should we createFakeParentDirectory for dst as it is the rename result?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 312420)
Time Spent: 0.5h (was: 20m)
> Rename does not preserve non-explicitly created interim directories
> ---
>
> Key: HDDS-2114
> URL: https://issues.apache.org/jira/browse/HDDS-2114
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Istvan Fajth
>Assignee: Lokesh Jain
>Priority: Critical
> Labels: pull-request-available
> Attachments: demonstrative_test.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> I am attaching a patch that adds a test that demonstrates the problem.
> The scenario is coming from the way how Hive implements acid transactions
> with the ORC table format, but the test is redacted to the simplest possible
> code that reproduces the issue.
> The scenario:
> * Given a 3 level directory structure, where the top level directory was
> explicitly created, and the interim directory is implicitly created (for
> example either by creating a file with create("/top/interim/file") or by
> creating a directory with mkdirs("top/interim/dir"))
> * When the leaf is moved out from the implicitly created directory making
> this directory an empty directory
> * Then a FileNotFoundException is thrown when getFileStatus or listStatus is
> called on the interim directory.
> The expected behaviour:
> after the directory is becoming empty, the directory should still be part of
> the file system, moreover an empty FileStatus array should be returned when
> listStatus is called on it, and also a valid FileStatus object should be
> returned when getFileStatus is called on it.
>
>
> As this issue is present with Hive, and as this is how a FileSystem is
> expected to work this seems to be an at least critical issue as I see, please
> feel free to change the priority if needed.
> Also please note that, if the interim directory is explicitly created with
> mkdirs("top/interim") before creating the leaf, then the issue does not
> appear.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-2114) Rename does not preserve non-explicitly created interim directories
[ https://issues.apache.org/jira/browse/HDDS-2114?focusedWorklogId=312103&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-312103 ] ASF GitHub Bot logged work on HDDS-2114: Author: ASF GitHub Bot Created on: 13/Sep/19 14:56 Start Date: 13/Sep/19 14:56 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #1440: HDDS-2114: Rename does not preserve non-explicitly created interim directories URL: https://github.com/apache/hadoop/pull/1440#issuecomment-531270620 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 36 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | -1 | mvninstall | 31 | hadoop-ozone in trunk failed. | | -1 | compile | 19 | hadoop-ozone in trunk failed. | | +1 | checkstyle | 64 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 896 | branch has no errors when building and testing our client artifacts. | | -1 | javadoc | 13 | hadoop-hdds in trunk failed. | | -1 | javadoc | 13 | hadoop-ozone in trunk failed. | | 0 | spotbugs | 159 | Used deprecated FindBugs config; considering switching to SpotBugs. | | -1 | findbugs | 22 | hadoop-ozone in trunk failed. | ||| _ Patch Compile Tests _ | | -1 | mvninstall | 30 | hadoop-ozone in the patch failed. | | -1 | compile | 20 | hadoop-ozone in the patch failed. | | -1 | javac | 20 | hadoop-ozone in the patch failed. | | -0 | checkstyle | 26 | hadoop-ozone: The patch generated 2 new + 121 unchanged - 0 fixed = 123 total (was 121) | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 716 | patch has no errors when building and testing our client artifacts. | | -1 | javadoc | 13 | hadoop-hdds in the patch failed. | | -1 | javadoc | 13 | hadoop-ozone in the patch failed. | | -1 | findbugs | 21 | hadoop-ozone in the patch failed. | ||| _ Other Tests _ | | -1 | unit | 137 | hadoop-hdds in the patch failed. | | -1 | unit | 24 | hadoop-ozone in the patch failed. | | +1 | asflicense | 28 | The patch does not generate ASF License warnings. | | | | 2860 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.container.keyvalue.TestKeyValueContainer | | | hadoop.ozone.container.ozoneimpl.TestOzoneContainer | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1440/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1440 | | JIRA Issue | HDDS-2114 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux c3b727f210a2 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 39e82ac | | Default Java | 1.8.0_222 | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1440/1/artifact/out/branch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1440/1/artifact/out/branch-compile-hadoop-ozone.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1440/1/artifact/out/branch-javadoc-hadoop-hdds.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1440/1/artifact/out/branch-javadoc-hadoop-ozone.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1440/1/artifact/out/branch-findbugs-hadoop-ozone.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1440/1/artifact/out/patch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1440/1/artifact/out/patch-compile-hadoop-ozone.txt | | javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1440/1/artifact/out/patch-compile-hadoop-ozone.txt | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1440/1/artifact/out/diff-checkstyle-hadoop-ozone.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1440/1/artifact/out/patch-javadoc-hadoop-hdds.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1440/1/artifact/out/patch-javadoc-hadoop-ozone.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1
[jira] [Work logged] (HDDS-2114) Rename does not preserve non-explicitly created interim directories
[
https://issues.apache.org/jira/browse/HDDS-2114?focusedWorklogId=312055&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-312055
]
ASF GitHub Bot logged work on HDDS-2114:
Author: ASF GitHub Bot
Created on: 13/Sep/19 14:07
Start Date: 13/Sep/19 14:07
Worklog Time Spent: 10m
Work Description: lokeshj1703 commented on pull request #1440: HDDS-2114:
Rename does not preserve non-explicitly created interim directories
URL: https://github.com/apache/hadoop/pull/1440
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 312055)
Remaining Estimate: 0h
Time Spent: 10m
> Rename does not preserve non-explicitly created interim directories
> ---
>
> Key: HDDS-2114
> URL: https://issues.apache.org/jira/browse/HDDS-2114
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Istvan Fajth
>Assignee: Lokesh Jain
>Priority: Critical
> Labels: pull-request-available
> Attachments: demonstrative_test.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> I am attaching a patch that adds a test that demonstrates the problem.
> The scenario is coming from the way how Hive implements acid transactions
> with the ORC table format, but the test is redacted to the simplest possible
> code that reproduces the issue.
> The scenario:
> * Given a 3 level directory structure, where the top level directory was
> explicitly created, and the interim directory is implicitly created (for
> example either by creating a file with create("/top/interim/file") or by
> creating a directory with mkdirs("top/interim/dir"))
> * When the leaf is moved out from the implicitly created directory making
> this directory an empty directory
> * Then a FileNotFoundException is thrown when getFileStatus or listStatus is
> called on the interim directory.
> The expected behaviour:
> after the directory is becoming empty, the directory should still be part of
> the file system, moreover an empty FileStatus array should be returned when
> listStatus is called on it, and also a valid FileStatus object should be
> returned when getFileStatus is called on it.
>
>
> As this issue is present with Hive, and as this is how a FileSystem is
> expected to work this seems to be an at least critical issue as I see, please
> feel free to change the priority if needed.
> Also please note that, if the interim directory is explicitly created with
> mkdirs("top/interim") before creating the leaf, then the issue does not
> appear.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
