[jira] [Work logged] (HDFS-15532) listFiles on root/InternalDir will fail if fallback root has file
[
https://issues.apache.org/jira/browse/HDFS-15532?focusedWorklogId=483596=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-483596
]
ASF GitHub Bot logged work on HDFS-15532:
-
Author: ASF GitHub Bot
Created on: 13/Sep/20 00:06
Start Date: 13/Sep/20 00:06
Worklog Time Spent: 10m
Work Description: umamaheswararao merged pull request #2298:
URL: https://github.com/apache/hadoop/pull/2298
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 483596)
Time Spent: 2h 40m (was: 2.5h)
> listFiles on root/InternalDir will fail if fallback root has file
> -
>
> Key: HDFS-15532
> URL: https://issues.apache.org/jira/browse/HDFS-15532
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.4.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> listFiles implementation gets the RemoteIterator created in
> InternalViewFSDirFs as the root is an InternalViewFSDir.
> If there is a fallback and a file exist at root level, it would have
> collected when collecting locatedStatuses.
> When its iterating over to that fallbacks file from RemoteIterator (which
> was returned from InternalViewFSDirFs ), iterator's next will will call
> getFileBlockLocations if it's a file.
> {code:java}
> @Override
> public LocatedFileStatus next() throws IOException {
> System.out.println(this);
> if (!hasNext()) {
> throw new NoSuchElementException("No more entries in " + f);
> }
> FileStatus result = stats[i++];
> // for files, use getBlockLocations(FileStatus, int, int) to avoid
> // calling getFileStatus(Path) to load the FileStatus again
> BlockLocation[] locs = result.isFile() ?
> getFileBlockLocations(result, 0, result.getLen()) :
> null;
> return new LocatedFileStatus(result, locs);
> }{code}
>
> this getFileBlockLocations will be made on InternalViewFSDirFs, as that
> Iterator created originally from that fs.
> InternalViewFSDirFs#getFileBlockLocations does not handle fallback cases.
> It's always expecting "/", this means it always assuming the dir.
> But with the fallback and returning Iterator from InternalViewFSDirFs, will
> create problems.
> Probably we need to handle fallback case in getFileBlockLocations as well.(
> Fallback only should be the reason for call coming to InternalViewFSDirFs
> with other than "/")
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15532) listFiles on root/InternalDir will fail if fallback root has file
[
https://issues.apache.org/jira/browse/HDFS-15532?focusedWorklogId=483504=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-483504
]
ASF GitHub Bot logged work on HDFS-15532:
-
Author: ASF GitHub Bot
Created on: 12/Sep/20 22:01
Start Date: 12/Sep/20 22:01
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2298:
URL: https://github.com/apache/hadoop/pull/2298#issuecomment-690996124
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 483504)
Time Spent: 2.5h (was: 2h 20m)
> listFiles on root/InternalDir will fail if fallback root has file
> -
>
> Key: HDFS-15532
> URL: https://issues.apache.org/jira/browse/HDFS-15532
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.4.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> listFiles implementation gets the RemoteIterator created in
> InternalViewFSDirFs as the root is an InternalViewFSDir.
> If there is a fallback and a file exist at root level, it would have
> collected when collecting locatedStatuses.
> When its iterating over to that fallbacks file from RemoteIterator (which
> was returned from InternalViewFSDirFs ), iterator's next will will call
> getFileBlockLocations if it's a file.
> {code:java}
> @Override
> public LocatedFileStatus next() throws IOException {
> System.out.println(this);
> if (!hasNext()) {
> throw new NoSuchElementException("No more entries in " + f);
> }
> FileStatus result = stats[i++];
> // for files, use getBlockLocations(FileStatus, int, int) to avoid
> // calling getFileStatus(Path) to load the FileStatus again
> BlockLocation[] locs = result.isFile() ?
> getFileBlockLocations(result, 0, result.getLen()) :
> null;
> return new LocatedFileStatus(result, locs);
> }{code}
>
> this getFileBlockLocations will be made on InternalViewFSDirFs, as that
> Iterator created originally from that fs.
> InternalViewFSDirFs#getFileBlockLocations does not handle fallback cases.
> It's always expecting "/", this means it always assuming the dir.
> But with the fallback and returning Iterator from InternalViewFSDirFs, will
> create problems.
> Probably we need to handle fallback case in getFileBlockLocations as well.(
> Fallback only should be the reason for call coming to InternalViewFSDirFs
> with other than "/")
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15532) listFiles on root/InternalDir will fail if fallback root has file
[
https://issues.apache.org/jira/browse/HDFS-15532?focusedWorklogId=483435=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-483435
]
ASF GitHub Bot logged work on HDFS-15532:
-
Author: ASF GitHub Bot
Created on: 12/Sep/20 21:55
Start Date: 12/Sep/20 21:55
Worklog Time Spent: 10m
Work Description: umamaheswararao commented on a change in pull request
#2298:
URL: https://github.com/apache/hadoop/pull/2298#discussion_r487375686
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
##
@@ -1263,6 +1263,26 @@ public boolean delete(final Path f)
public BlockLocation[] getFileBlockLocations(final FileStatus fs,
final long start, final long len) throws
FileNotFoundException, IOException {
+
+ // When application calls listFiles on internalDir, it would return
+ // RemoteIterator from InternalDirOfViewFs. If there is a fallBack, there
+ // is a chance of files exists under that internalDir in fallback.
+ // Iterator#next will call getFileBlockLocations with that files. So, we
+ // should return getFileBlockLocations on fallback. See HDFS-15532.
+ if (fs.getPath() != InodeTree.SlashPath && this.fsState
+ .getRootFallbackLink() != null) {
+FileSystem linkedFallbackFs =
+this.fsState.getRootFallbackLink().getTargetFileSystem();
+Path parent = Path.getPathWithoutSchemeAndAuthority(
+new Path(theInternalDir.fullPath));
+String leafChild = (InodeTree.SlashPath.equals(fs.getPath())) ?
+InodeTree.SlashPath.toString() :
+fs.getPath().getName();
Review comment:
Thanks @ayushtkn for review! I have removed this redundant condition.
I also realized this bug exist in ViewFs as well. I fixed and added a test
case for it. Thanks
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
##
@@ -1263,6 +1263,26 @@ public boolean delete(final Path f)
public BlockLocation[] getFileBlockLocations(final FileStatus fs,
final long start, final long len) throws
FileNotFoundException, IOException {
+
+ // When application calls listFiles on internalDir, it would return
+ // RemoteIterator from InternalDirOfViewFs. If there is a fallBack, there
+ // is a chance of files exists under that internalDir in fallback.
+ // Iterator#next will call getFileBlockLocations with that files. So, we
+ // should return getFileBlockLocations on fallback. See HDFS-15532.
+ if (fs.getPath() != InodeTree.SlashPath && this.fsState
+ .getRootFallbackLink() != null) {
+FileSystem linkedFallbackFs =
+this.fsState.getRootFallbackLink().getTargetFileSystem();
+Path parent = Path.getPathWithoutSchemeAndAuthority(
+new Path(theInternalDir.fullPath));
+String leafChild = (InodeTree.SlashPath.equals(fs.getPath())) ?
+InodeTree.SlashPath.toString() :
+fs.getPath().getName();
Review comment:
Thanks @ayushtkn for review! I have removed this redundant condition.
I realized that, this bug exist in ViewFs as well. I fixed and added a test
case for it. Thanks
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
##
@@ -1263,6 +1263,26 @@ public boolean delete(final Path f)
public BlockLocation[] getFileBlockLocations(final FileStatus fs,
final long start, final long len) throws
FileNotFoundException, IOException {
+
+ // When application calls listFiles on internalDir, it would return
+ // RemoteIterator from InternalDirOfViewFs. If there is a fallBack, there
+ // is a chance of files exists under that internalDir in fallback.
+ // Iterator#next will call getFileBlockLocations with that files. So, we
+ // should return getFileBlockLocations on fallback. See HDFS-15532.
+ if (fs.getPath() != InodeTree.SlashPath && this.fsState
+ .getRootFallbackLink() != null) {
+FileSystem linkedFallbackFs =
+this.fsState.getRootFallbackLink().getTargetFileSystem();
+Path parent = Path.getPathWithoutSchemeAndAuthority(
+new Path(theInternalDir.fullPath));
+String leafChild = (InodeTree.SlashPath.equals(fs.getPath())) ?
+InodeTree.SlashPath.toString() :
+fs.getPath().getName();
Review comment:
Thanks @ayushtkn for review! I have removed this redundant condition.
I also realized this bug exist in ViewFs as well. I fixed and added a test
case for it. Thanks
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
##
@@ -1263,6 +1263,26 @@ public
[jira] [Work logged] (HDFS-15532) listFiles on root/InternalDir will fail if fallback root has file
[
https://issues.apache.org/jira/browse/HDFS-15532?focusedWorklogId=483179=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-483179
]
ASF GitHub Bot logged work on HDFS-15532:
-
Author: ASF GitHub Bot
Created on: 12/Sep/20 21:27
Start Date: 12/Sep/20 21:27
Worklog Time Spent: 10m
Work Description: ayushtkn commented on a change in pull request #2298:
URL: https://github.com/apache/hadoop/pull/2298#discussion_r487368520
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
##
@@ -1263,6 +1263,26 @@ public boolean delete(final Path f)
public BlockLocation[] getFileBlockLocations(final FileStatus fs,
final long start, final long len) throws
FileNotFoundException, IOException {
+
+ // When application calls listFiles on internalDir, it would return
+ // RemoteIterator from InternalDirOfViewFs. If there is a fallBack, there
+ // is a chance of files exists under that internalDir in fallback.
+ // Iterator#next will call getFileBlockLocations with that files. So, we
+ // should return getFileBlockLocations on fallback. See HDFS-15532.
+ if (fs.getPath() != InodeTree.SlashPath && this.fsState
+ .getRootFallbackLink() != null) {
+FileSystem linkedFallbackFs =
+this.fsState.getRootFallbackLink().getTargetFileSystem();
+Path parent = Path.getPathWithoutSchemeAndAuthority(
+new Path(theInternalDir.fullPath));
+String leafChild = (InodeTree.SlashPath.equals(fs.getPath())) ?
+InodeTree.SlashPath.toString() :
+fs.getPath().getName();
Review comment:
`(InodeTree.SlashPath.equals(fs.getPath()))`
In which case this would be true? Provided there is a check in the if
condition
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
##
@@ -1263,6 +1263,26 @@ public boolean delete(final Path f)
public BlockLocation[] getFileBlockLocations(final FileStatus fs,
final long start, final long len) throws
FileNotFoundException, IOException {
+
+ // When application calls listFiles on internalDir, it would return
+ // RemoteIterator from InternalDirOfViewFs. If there is a fallBack, there
+ // is a chance of files exists under that internalDir in fallback.
+ // Iterator#next will call getFileBlockLocations with that files. So, we
+ // should return getFileBlockLocations on fallback. See HDFS-15532.
+ if (fs.getPath() != InodeTree.SlashPath && this.fsState
+ .getRootFallbackLink() != null) {
+FileSystem linkedFallbackFs =
+this.fsState.getRootFallbackLink().getTargetFileSystem();
+Path parent = Path.getPathWithoutSchemeAndAuthority(
+new Path(theInternalDir.fullPath));
+String leafChild = (InodeTree.SlashPath.equals(fs.getPath())) ?
+InodeTree.SlashPath.toString() :
+fs.getPath().getName();
Review comment:
`(InodeTree.SlashPath.equals(fs.getPath()))`
In which case this would be true? Provided there is a check in the if
condition
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
##
@@ -1263,6 +1263,26 @@ public boolean delete(final Path f)
public BlockLocation[] getFileBlockLocations(final FileStatus fs,
final long start, final long len) throws
FileNotFoundException, IOException {
+
+ // When application calls listFiles on internalDir, it would return
+ // RemoteIterator from InternalDirOfViewFs. If there is a fallBack, there
+ // is a chance of files exists under that internalDir in fallback.
+ // Iterator#next will call getFileBlockLocations with that files. So, we
+ // should return getFileBlockLocations on fallback. See HDFS-15532.
+ if (fs.getPath() != InodeTree.SlashPath && this.fsState
+ .getRootFallbackLink() != null) {
+FileSystem linkedFallbackFs =
+this.fsState.getRootFallbackLink().getTargetFileSystem();
+Path parent = Path.getPathWithoutSchemeAndAuthority(
+new Path(theInternalDir.fullPath));
+String leafChild = (InodeTree.SlashPath.equals(fs.getPath())) ?
+InodeTree.SlashPath.toString() :
+fs.getPath().getName();
Review comment:
`(InodeTree.SlashPath.equals(fs.getPath()))`
In which case this would be true? Provided there is a check in the if
condition
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
##
@@ -1263,6 +1263,26 @@ public boolean delete(final Path f)
public BlockLocation[] getFileBlockLocations(final FileStatus fs,
final long start, final
[jira] [Work logged] (HDFS-15532) listFiles on root/InternalDir will fail if fallback root has file
[
https://issues.apache.org/jira/browse/HDFS-15532?focusedWorklogId=483073=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-483073
]
ASF GitHub Bot logged work on HDFS-15532:
-
Author: ASF GitHub Bot
Created on: 12/Sep/20 20:52
Start Date: 12/Sep/20 20:52
Worklog Time Spent: 10m
Work Description: umamaheswararao commented on a change in pull request
#2298:
URL: https://github.com/apache/hadoop/pull/2298#discussion_r487375686
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
##
@@ -1263,6 +1263,26 @@ public boolean delete(final Path f)
public BlockLocation[] getFileBlockLocations(final FileStatus fs,
final long start, final long len) throws
FileNotFoundException, IOException {
+
+ // When application calls listFiles on internalDir, it would return
+ // RemoteIterator from InternalDirOfViewFs. If there is a fallBack, there
+ // is a chance of files exists under that internalDir in fallback.
+ // Iterator#next will call getFileBlockLocations with that files. So, we
+ // should return getFileBlockLocations on fallback. See HDFS-15532.
+ if (fs.getPath() != InodeTree.SlashPath && this.fsState
+ .getRootFallbackLink() != null) {
+FileSystem linkedFallbackFs =
+this.fsState.getRootFallbackLink().getTargetFileSystem();
+Path parent = Path.getPathWithoutSchemeAndAuthority(
+new Path(theInternalDir.fullPath));
+String leafChild = (InodeTree.SlashPath.equals(fs.getPath())) ?
+InodeTree.SlashPath.toString() :
+fs.getPath().getName();
Review comment:
Thanks @ayushtkn for review! I have removed this redundant condition.
I also realized this bug exist in ViewFs as well. I fixed and added a test
case for it. Thanks
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
##
@@ -1263,6 +1263,26 @@ public boolean delete(final Path f)
public BlockLocation[] getFileBlockLocations(final FileStatus fs,
final long start, final long len) throws
FileNotFoundException, IOException {
+
+ // When application calls listFiles on internalDir, it would return
+ // RemoteIterator from InternalDirOfViewFs. If there is a fallBack, there
+ // is a chance of files exists under that internalDir in fallback.
+ // Iterator#next will call getFileBlockLocations with that files. So, we
+ // should return getFileBlockLocations on fallback. See HDFS-15532.
+ if (fs.getPath() != InodeTree.SlashPath && this.fsState
+ .getRootFallbackLink() != null) {
+FileSystem linkedFallbackFs =
+this.fsState.getRootFallbackLink().getTargetFileSystem();
+Path parent = Path.getPathWithoutSchemeAndAuthority(
+new Path(theInternalDir.fullPath));
+String leafChild = (InodeTree.SlashPath.equals(fs.getPath())) ?
+InodeTree.SlashPath.toString() :
+fs.getPath().getName();
Review comment:
Thanks @ayushtkn for review! I have removed this redundant condition.
I realized that, this bug exist in ViewFs as well. I fixed and added a test
case for it. Thanks
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
##
@@ -1263,6 +1263,26 @@ public boolean delete(final Path f)
public BlockLocation[] getFileBlockLocations(final FileStatus fs,
final long start, final long len) throws
FileNotFoundException, IOException {
+
+ // When application calls listFiles on internalDir, it would return
+ // RemoteIterator from InternalDirOfViewFs. If there is a fallBack, there
+ // is a chance of files exists under that internalDir in fallback.
+ // Iterator#next will call getFileBlockLocations with that files. So, we
+ // should return getFileBlockLocations on fallback. See HDFS-15532.
+ if (fs.getPath() != InodeTree.SlashPath && this.fsState
+ .getRootFallbackLink() != null) {
+FileSystem linkedFallbackFs =
+this.fsState.getRootFallbackLink().getTargetFileSystem();
+Path parent = Path.getPathWithoutSchemeAndAuthority(
+new Path(theInternalDir.fullPath));
+String leafChild = (InodeTree.SlashPath.equals(fs.getPath())) ?
+InodeTree.SlashPath.toString() :
+fs.getPath().getName();
Review comment:
Thanks @ayushtkn for review! I have removed this redundant condition.
I also realized this bug exist in ViewFs as well. I fixed and added a test
case for it. Thanks
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
##
@@ -1263,6 +1263,26 @@ public
[jira] [Work logged] (HDFS-15532) listFiles on root/InternalDir will fail if fallback root has file
[
https://issues.apache.org/jira/browse/HDFS-15532?focusedWorklogId=482851=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-482851
]
ASF GitHub Bot logged work on HDFS-15532:
-
Author: ASF GitHub Bot
Created on: 12/Sep/20 20:27
Start Date: 12/Sep/20 20:27
Worklog Time Spent: 10m
Work Description: umamaheswararao opened a new pull request #2298:
URL: https://github.com/apache/hadoop/pull/2298
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 482851)
Time Spent: 1h 50m (was: 1h 40m)
> listFiles on root/InternalDir will fail if fallback root has file
> -
>
> Key: HDFS-15532
> URL: https://issues.apache.org/jira/browse/HDFS-15532
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.4.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> listFiles implementation gets the RemoteIterator created in
> InternalViewFSDirFs as the root is an InternalViewFSDir.
> If there is a fallback and a file exist at root level, it would have
> collected when collecting locatedStatuses.
> When its iterating over to that fallbacks file from RemoteIterator (which
> was returned from InternalViewFSDirFs ), iterator's next will will call
> getFileBlockLocations if it's a file.
> {code:java}
> @Override
> public LocatedFileStatus next() throws IOException {
> System.out.println(this);
> if (!hasNext()) {
> throw new NoSuchElementException("No more entries in " + f);
> }
> FileStatus result = stats[i++];
> // for files, use getBlockLocations(FileStatus, int, int) to avoid
> // calling getFileStatus(Path) to load the FileStatus again
> BlockLocation[] locs = result.isFile() ?
> getFileBlockLocations(result, 0, result.getLen()) :
> null;
> return new LocatedFileStatus(result, locs);
> }{code}
>
> this getFileBlockLocations will be made on InternalViewFSDirFs, as that
> Iterator created originally from that fs.
> InternalViewFSDirFs#getFileBlockLocations does not handle fallback cases.
> It's always expecting "/", this means it always assuming the dir.
> But with the fallback and returning Iterator from InternalViewFSDirFs, will
> create problems.
> Probably we need to handle fallback case in getFileBlockLocations as well.(
> Fallback only should be the reason for call coming to InternalViewFSDirFs
> with other than "/")
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15532) listFiles on root/InternalDir will fail if fallback root has file
[
https://issues.apache.org/jira/browse/HDFS-15532?focusedWorklogId=482770=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-482770
]
ASF GitHub Bot logged work on HDFS-15532:
-
Author: ASF GitHub Bot
Created on: 12/Sep/20 20:19
Start Date: 12/Sep/20 20:19
Worklog Time Spent: 10m
Work Description: ayushtkn commented on a change in pull request #2298:
URL: https://github.com/apache/hadoop/pull/2298#discussion_r487368520
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
##
@@ -1263,6 +1263,26 @@ public boolean delete(final Path f)
public BlockLocation[] getFileBlockLocations(final FileStatus fs,
final long start, final long len) throws
FileNotFoundException, IOException {
+
+ // When application calls listFiles on internalDir, it would return
+ // RemoteIterator from InternalDirOfViewFs. If there is a fallBack, there
+ // is a chance of files exists under that internalDir in fallback.
+ // Iterator#next will call getFileBlockLocations with that files. So, we
+ // should return getFileBlockLocations on fallback. See HDFS-15532.
+ if (fs.getPath() != InodeTree.SlashPath && this.fsState
+ .getRootFallbackLink() != null) {
+FileSystem linkedFallbackFs =
+this.fsState.getRootFallbackLink().getTargetFileSystem();
+Path parent = Path.getPathWithoutSchemeAndAuthority(
+new Path(theInternalDir.fullPath));
+String leafChild = (InodeTree.SlashPath.equals(fs.getPath())) ?
+InodeTree.SlashPath.toString() :
+fs.getPath().getName();
Review comment:
`(InodeTree.SlashPath.equals(fs.getPath()))`
In which case this would be true? Provided there is a check in the if
condition
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 482770)
Time Spent: 1h 40m (was: 1.5h)
> listFiles on root/InternalDir will fail if fallback root has file
> -
>
> Key: HDFS-15532
> URL: https://issues.apache.org/jira/browse/HDFS-15532
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.4.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> listFiles implementation gets the RemoteIterator created in
> InternalViewFSDirFs as the root is an InternalViewFSDir.
> If there is a fallback and a file exist at root level, it would have
> collected when collecting locatedStatuses.
> When its iterating over to that fallbacks file from RemoteIterator (which
> was returned from InternalViewFSDirFs ), iterator's next will will call
> getFileBlockLocations if it's a file.
> {code:java}
> @Override
> public LocatedFileStatus next() throws IOException {
> System.out.println(this);
> if (!hasNext()) {
> throw new NoSuchElementException("No more entries in " + f);
> }
> FileStatus result = stats[i++];
> // for files, use getBlockLocations(FileStatus, int, int) to avoid
> // calling getFileStatus(Path) to load the FileStatus again
> BlockLocation[] locs = result.isFile() ?
> getFileBlockLocations(result, 0, result.getLen()) :
> null;
> return new LocatedFileStatus(result, locs);
> }{code}
>
> this getFileBlockLocations will be made on InternalViewFSDirFs, as that
> Iterator created originally from that fs.
> InternalViewFSDirFs#getFileBlockLocations does not handle fallback cases.
> It's always expecting "/", this means it always assuming the dir.
> But with the fallback and returning Iterator from InternalViewFSDirFs, will
> create problems.
> Probably we need to handle fallback case in getFileBlockLocations as well.(
> Fallback only should be the reason for call coming to InternalViewFSDirFs
> with other than "/")
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15532) listFiles on root/InternalDir will fail if fallback root has file
[
https://issues.apache.org/jira/browse/HDFS-15532?focusedWorklogId=482683=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-482683
]
ASF GitHub Bot logged work on HDFS-15532:
-
Author: ASF GitHub Bot
Created on: 12/Sep/20 20:12
Start Date: 12/Sep/20 20:12
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2298:
URL: https://github.com/apache/hadoop/pull/2298#issuecomment-690996124
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 482683)
Time Spent: 1.5h (was: 1h 20m)
> listFiles on root/InternalDir will fail if fallback root has file
> -
>
> Key: HDFS-15532
> URL: https://issues.apache.org/jira/browse/HDFS-15532
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.4.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> listFiles implementation gets the RemoteIterator created in
> InternalViewFSDirFs as the root is an InternalViewFSDir.
> If there is a fallback and a file exist at root level, it would have
> collected when collecting locatedStatuses.
> When its iterating over to that fallbacks file from RemoteIterator (which
> was returned from InternalViewFSDirFs ), iterator's next will will call
> getFileBlockLocations if it's a file.
> {code:java}
> @Override
> public LocatedFileStatus next() throws IOException {
> System.out.println(this);
> if (!hasNext()) {
> throw new NoSuchElementException("No more entries in " + f);
> }
> FileStatus result = stats[i++];
> // for files, use getBlockLocations(FileStatus, int, int) to avoid
> // calling getFileStatus(Path) to load the FileStatus again
> BlockLocation[] locs = result.isFile() ?
> getFileBlockLocations(result, 0, result.getLen()) :
> null;
> return new LocatedFileStatus(result, locs);
> }{code}
>
> this getFileBlockLocations will be made on InternalViewFSDirFs, as that
> Iterator created originally from that fs.
> InternalViewFSDirFs#getFileBlockLocations does not handle fallback cases.
> It's always expecting "/", this means it always assuming the dir.
> But with the fallback and returning Iterator from InternalViewFSDirFs, will
> create problems.
> Probably we need to handle fallback case in getFileBlockLocations as well.(
> Fallback only should be the reason for call coming to InternalViewFSDirFs
> with other than "/")
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15532) listFiles on root/InternalDir will fail if fallback root has file
[
https://issues.apache.org/jira/browse/HDFS-15532?focusedWorklogId=482613=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-482613
]
ASF GitHub Bot logged work on HDFS-15532:
-
Author: ASF GitHub Bot
Created on: 12/Sep/20 20:06
Start Date: 12/Sep/20 20:06
Worklog Time Spent: 10m
Work Description: umamaheswararao commented on a change in pull request
#2298:
URL: https://github.com/apache/hadoop/pull/2298#discussion_r487375686
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
##
@@ -1263,6 +1263,26 @@ public boolean delete(final Path f)
public BlockLocation[] getFileBlockLocations(final FileStatus fs,
final long start, final long len) throws
FileNotFoundException, IOException {
+
+ // When application calls listFiles on internalDir, it would return
+ // RemoteIterator from InternalDirOfViewFs. If there is a fallBack, there
+ // is a chance of files exists under that internalDir in fallback.
+ // Iterator#next will call getFileBlockLocations with that files. So, we
+ // should return getFileBlockLocations on fallback. See HDFS-15532.
+ if (fs.getPath() != InodeTree.SlashPath && this.fsState
+ .getRootFallbackLink() != null) {
+FileSystem linkedFallbackFs =
+this.fsState.getRootFallbackLink().getTargetFileSystem();
+Path parent = Path.getPathWithoutSchemeAndAuthority(
+new Path(theInternalDir.fullPath));
+String leafChild = (InodeTree.SlashPath.equals(fs.getPath())) ?
+InodeTree.SlashPath.toString() :
+fs.getPath().getName();
Review comment:
Thanks @ayushtkn for review! I have removed this redundant condition.
I also realized this bug exist in ViewFs as well. I fixed and added a test
case for it. Thanks
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
##
@@ -1263,6 +1263,26 @@ public boolean delete(final Path f)
public BlockLocation[] getFileBlockLocations(final FileStatus fs,
final long start, final long len) throws
FileNotFoundException, IOException {
+
+ // When application calls listFiles on internalDir, it would return
+ // RemoteIterator from InternalDirOfViewFs. If there is a fallBack, there
+ // is a chance of files exists under that internalDir in fallback.
+ // Iterator#next will call getFileBlockLocations with that files. So, we
+ // should return getFileBlockLocations on fallback. See HDFS-15532.
+ if (fs.getPath() != InodeTree.SlashPath && this.fsState
+ .getRootFallbackLink() != null) {
+FileSystem linkedFallbackFs =
+this.fsState.getRootFallbackLink().getTargetFileSystem();
+Path parent = Path.getPathWithoutSchemeAndAuthority(
+new Path(theInternalDir.fullPath));
+String leafChild = (InodeTree.SlashPath.equals(fs.getPath())) ?
+InodeTree.SlashPath.toString() :
+fs.getPath().getName();
Review comment:
Thanks @ayushtkn for review! I have removed this redundant condition.
I realized that, this bug exist in ViewFs as well. I fixed and added a test
case for it. Thanks
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 482613)
Time Spent: 1h 20m (was: 1h 10m)
> listFiles on root/InternalDir will fail if fallback root has file
> -
>
> Key: HDFS-15532
> URL: https://issues.apache.org/jira/browse/HDFS-15532
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.4.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> listFiles implementation gets the RemoteIterator created in
> InternalViewFSDirFs as the root is an InternalViewFSDir.
> If there is a fallback and a file exist at root level, it would have
> collected when collecting locatedStatuses.
> When its iterating over to that fallbacks file from RemoteIterator (which
> was returned from InternalViewFSDirFs ), iterator's next will will call
> getFileBlockLocations if it's a file.
> {code:java}
> @Override
> public LocatedFileStatus next() throws IOException {
> System.out.println(this);
> if (!hasNext()) {
> throw new
[jira] [Work logged] (HDFS-15532) listFiles on root/InternalDir will fail if fallback root has file
[ https://issues.apache.org/jira/browse/HDFS-15532?focusedWorklogId=482479=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-482479 ] ASF GitHub Bot logged work on HDFS-15532: - Author: ASF GitHub Bot Created on: 12/Sep/20 13:04 Start Date: 12/Sep/20 13:04 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #2298: URL: https://github.com/apache/hadoop/pull/2298#issuecomment-691484894 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 21s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. | ||| _ trunk Compile Tests _ | | +0 :ok: | mvndep | 3m 30s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 33m 58s | trunk passed | | +1 :green_heart: | compile | 29m 32s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 23m 55s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 3m 35s | trunk passed | | +1 :green_heart: | mvnsite | 3m 39s | trunk passed | | +1 :green_heart: | shadedclient | 24m 48s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 1m 48s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 3m 32s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 4m 10s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 7m 3s | trunk passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 30s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 45s | the patch passed | | +1 :green_heart: | compile | 31m 23s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javac | 31m 23s | the patch passed | | +1 :green_heart: | compile | 27m 5s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | javac | 27m 5s | the patch passed | | +1 :green_heart: | checkstyle | 3m 52s | the patch passed | | +1 :green_heart: | mvnsite | 3m 58s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 19m 10s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 1m 53s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 3m 44s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | findbugs | 8m 0s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 13m 4s | hadoop-common in the patch passed. | | -1 :x: | unit | 144m 16s | hadoop-hdfs in the patch passed. | | +1 :green_heart: | asflicense | 1m 31s | The patch does not generate ASF License warnings. | | | | 396m 13s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hdfs.TestFileChecksumCompositeCrc | | | hadoop.hdfs.TestFileChecksum | | | hadoop.hdfs.server.datanode.TestBPOfferService | | | hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier | | | hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics | | | hadoop.hdfs.server.namenode.TestAddStripedBlockInFBR | | | hadoop.hdfs.TestSafeModeWithStripedFileWithRandomECPolicy | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2298/3/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2298 | | JIRA Issue | HDFS-15532 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 4037b09c1838 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / 958cab804ef | | Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | Multi-JDK versions |
[jira] [Work logged] (HDFS-15532) listFiles on root/InternalDir will fail if fallback root has file
[
https://issues.apache.org/jira/browse/HDFS-15532?focusedWorklogId=482436=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-482436
]
ASF GitHub Bot logged work on HDFS-15532:
-
Author: ASF GitHub Bot
Created on: 12/Sep/20 06:27
Start Date: 12/Sep/20 06:27
Worklog Time Spent: 10m
Work Description: umamaheswararao commented on a change in pull request
#2298:
URL: https://github.com/apache/hadoop/pull/2298#discussion_r487375686
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
##
@@ -1263,6 +1263,26 @@ public boolean delete(final Path f)
public BlockLocation[] getFileBlockLocations(final FileStatus fs,
final long start, final long len) throws
FileNotFoundException, IOException {
+
+ // When application calls listFiles on internalDir, it would return
+ // RemoteIterator from InternalDirOfViewFs. If there is a fallBack, there
+ // is a chance of files exists under that internalDir in fallback.
+ // Iterator#next will call getFileBlockLocations with that files. So, we
+ // should return getFileBlockLocations on fallback. See HDFS-15532.
+ if (fs.getPath() != InodeTree.SlashPath && this.fsState
+ .getRootFallbackLink() != null) {
+FileSystem linkedFallbackFs =
+this.fsState.getRootFallbackLink().getTargetFileSystem();
+Path parent = Path.getPathWithoutSchemeAndAuthority(
+new Path(theInternalDir.fullPath));
+String leafChild = (InodeTree.SlashPath.equals(fs.getPath())) ?
+InodeTree.SlashPath.toString() :
+fs.getPath().getName();
Review comment:
Thanks @ayushtkn for review! I have removed this redundant condition.
I also realized this bug exist in ViewFs as well. I fixed and added a test
case for it. Thanks
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 482436)
Time Spent: 50m (was: 40m)
> listFiles on root/InternalDir will fail if fallback root has file
> -
>
> Key: HDFS-15532
> URL: https://issues.apache.org/jira/browse/HDFS-15532
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.4.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> listFiles implementation gets the RemoteIterator created in
> InternalViewFSDirFs as the root is an InternalViewFSDir.
> If there is a fallback and a file exist at root level, it would have
> collected when collecting locatedStatuses.
> When its iterating over to that fallbacks file from RemoteIterator (which
> was returned from InternalViewFSDirFs ), iterator's next will will call
> getFileBlockLocations if it's a file.
> {code:java}
> @Override
> public LocatedFileStatus next() throws IOException {
> System.out.println(this);
> if (!hasNext()) {
> throw new NoSuchElementException("No more entries in " + f);
> }
> FileStatus result = stats[i++];
> // for files, use getBlockLocations(FileStatus, int, int) to avoid
> // calling getFileStatus(Path) to load the FileStatus again
> BlockLocation[] locs = result.isFile() ?
> getFileBlockLocations(result, 0, result.getLen()) :
> null;
> return new LocatedFileStatus(result, locs);
> }{code}
>
> this getFileBlockLocations will be made on InternalViewFSDirFs, as that
> Iterator created originally from that fs.
> InternalViewFSDirFs#getFileBlockLocations does not handle fallback cases.
> It's always expecting "/", this means it always assuming the dir.
> But with the fallback and returning Iterator from InternalViewFSDirFs, will
> create problems.
> Probably we need to handle fallback case in getFileBlockLocations as well.(
> Fallback only should be the reason for call coming to InternalViewFSDirFs
> with other than "/")
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15532) listFiles on root/InternalDir will fail if fallback root has file
[
https://issues.apache.org/jira/browse/HDFS-15532?focusedWorklogId=482438=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-482438
]
ASF GitHub Bot logged work on HDFS-15532:
-
Author: ASF GitHub Bot
Created on: 12/Sep/20 06:27
Start Date: 12/Sep/20 06:27
Worklog Time Spent: 10m
Work Description: umamaheswararao commented on a change in pull request
#2298:
URL: https://github.com/apache/hadoop/pull/2298#discussion_r487375686
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
##
@@ -1263,6 +1263,26 @@ public boolean delete(final Path f)
public BlockLocation[] getFileBlockLocations(final FileStatus fs,
final long start, final long len) throws
FileNotFoundException, IOException {
+
+ // When application calls listFiles on internalDir, it would return
+ // RemoteIterator from InternalDirOfViewFs. If there is a fallBack, there
+ // is a chance of files exists under that internalDir in fallback.
+ // Iterator#next will call getFileBlockLocations with that files. So, we
+ // should return getFileBlockLocations on fallback. See HDFS-15532.
+ if (fs.getPath() != InodeTree.SlashPath && this.fsState
+ .getRootFallbackLink() != null) {
+FileSystem linkedFallbackFs =
+this.fsState.getRootFallbackLink().getTargetFileSystem();
+Path parent = Path.getPathWithoutSchemeAndAuthority(
+new Path(theInternalDir.fullPath));
+String leafChild = (InodeTree.SlashPath.equals(fs.getPath())) ?
+InodeTree.SlashPath.toString() :
+fs.getPath().getName();
Review comment:
Thanks @ayushtkn for review! I have removed this redundant condition.
I realized that, this bug exist in ViewFs as well. I fixed and added a test
case for it. Thanks
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 482438)
Time Spent: 1h (was: 50m)
> listFiles on root/InternalDir will fail if fallback root has file
> -
>
> Key: HDFS-15532
> URL: https://issues.apache.org/jira/browse/HDFS-15532
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.4.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> listFiles implementation gets the RemoteIterator created in
> InternalViewFSDirFs as the root is an InternalViewFSDir.
> If there is a fallback and a file exist at root level, it would have
> collected when collecting locatedStatuses.
> When its iterating over to that fallbacks file from RemoteIterator (which
> was returned from InternalViewFSDirFs ), iterator's next will will call
> getFileBlockLocations if it's a file.
> {code:java}
> @Override
> public LocatedFileStatus next() throws IOException {
> System.out.println(this);
> if (!hasNext()) {
> throw new NoSuchElementException("No more entries in " + f);
> }
> FileStatus result = stats[i++];
> // for files, use getBlockLocations(FileStatus, int, int) to avoid
> // calling getFileStatus(Path) to load the FileStatus again
> BlockLocation[] locs = result.isFile() ?
> getFileBlockLocations(result, 0, result.getLen()) :
> null;
> return new LocatedFileStatus(result, locs);
> }{code}
>
> this getFileBlockLocations will be made on InternalViewFSDirFs, as that
> Iterator created originally from that fs.
> InternalViewFSDirFs#getFileBlockLocations does not handle fallback cases.
> It's always expecting "/", this means it always assuming the dir.
> But with the fallback and returning Iterator from InternalViewFSDirFs, will
> create problems.
> Probably we need to handle fallback case in getFileBlockLocations as well.(
> Fallback only should be the reason for call coming to InternalViewFSDirFs
> with other than "/")
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15532) listFiles on root/InternalDir will fail if fallback root has file
[
https://issues.apache.org/jira/browse/HDFS-15532?focusedWorklogId=482425=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-482425
]
ASF GitHub Bot logged work on HDFS-15532:
-
Author: ASF GitHub Bot
Created on: 12/Sep/20 04:58
Start Date: 12/Sep/20 04:58
Worklog Time Spent: 10m
Work Description: ayushtkn commented on a change in pull request #2298:
URL: https://github.com/apache/hadoop/pull/2298#discussion_r487368520
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
##
@@ -1263,6 +1263,26 @@ public boolean delete(final Path f)
public BlockLocation[] getFileBlockLocations(final FileStatus fs,
final long start, final long len) throws
FileNotFoundException, IOException {
+
+ // When application calls listFiles on internalDir, it would return
+ // RemoteIterator from InternalDirOfViewFs. If there is a fallBack, there
+ // is a chance of files exists under that internalDir in fallback.
+ // Iterator#next will call getFileBlockLocations with that files. So, we
+ // should return getFileBlockLocations on fallback. See HDFS-15532.
+ if (fs.getPath() != InodeTree.SlashPath && this.fsState
+ .getRootFallbackLink() != null) {
+FileSystem linkedFallbackFs =
+this.fsState.getRootFallbackLink().getTargetFileSystem();
+Path parent = Path.getPathWithoutSchemeAndAuthority(
+new Path(theInternalDir.fullPath));
+String leafChild = (InodeTree.SlashPath.equals(fs.getPath())) ?
+InodeTree.SlashPath.toString() :
+fs.getPath().getName();
Review comment:
`(InodeTree.SlashPath.equals(fs.getPath()))`
In which case this would be true? Provided there is a check in the if
condition
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 482425)
Time Spent: 40m (was: 0.5h)
> listFiles on root/InternalDir will fail if fallback root has file
> -
>
> Key: HDFS-15532
> URL: https://issues.apache.org/jira/browse/HDFS-15532
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.4.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> listFiles implementation gets the RemoteIterator created in
> InternalViewFSDirFs as the root is an InternalViewFSDir.
> If there is a fallback and a file exist at root level, it would have
> collected when collecting locatedStatuses.
> When its iterating over to that fallbacks file from RemoteIterator (which
> was returned from InternalViewFSDirFs ), iterator's next will will call
> getFileBlockLocations if it's a file.
> {code:java}
> @Override
> public LocatedFileStatus next() throws IOException {
> System.out.println(this);
> if (!hasNext()) {
> throw new NoSuchElementException("No more entries in " + f);
> }
> FileStatus result = stats[i++];
> // for files, use getBlockLocations(FileStatus, int, int) to avoid
> // calling getFileStatus(Path) to load the FileStatus again
> BlockLocation[] locs = result.isFile() ?
> getFileBlockLocations(result, 0, result.getLen()) :
> null;
> return new LocatedFileStatus(result, locs);
> }{code}
>
> this getFileBlockLocations will be made on InternalViewFSDirFs, as that
> Iterator created originally from that fs.
> InternalViewFSDirFs#getFileBlockLocations does not handle fallback cases.
> It's always expecting "/", this means it always assuming the dir.
> But with the fallback and returning Iterator from InternalViewFSDirFs, will
> create problems.
> Probably we need to handle fallback case in getFileBlockLocations as well.(
> Fallback only should be the reason for call coming to InternalViewFSDirFs
> with other than "/")
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15532) listFiles on root/InternalDir will fail if fallback root has file
[ https://issues.apache.org/jira/browse/HDFS-15532?focusedWorklogId=482376=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-482376 ] ASF GitHub Bot logged work on HDFS-15532: - Author: ASF GitHub Bot Created on: 11/Sep/20 23:34 Start Date: 11/Sep/20 23:34 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #2298: URL: https://github.com/apache/hadoop/pull/2298#issuecomment-691354335 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 13s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | +0 :ok: | mvndep | 3m 20s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 29m 13s | trunk passed | | +1 :green_heart: | compile | 22m 44s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 23m 1s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 3m 27s | trunk passed | | +1 :green_heart: | mvnsite | 3m 28s | trunk passed | | +1 :green_heart: | shadedclient | 25m 50s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 1m 42s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 3m 26s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 3m 56s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 6m 52s | trunk passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 25s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 33s | the patch passed | | +1 :green_heart: | compile | 26m 25s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javac | 26m 25s | the patch passed | | +1 :green_heart: | compile | 21m 39s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | javac | 21m 39s | the patch passed | | +1 :green_heart: | checkstyle | 3m 27s | the patch passed | | +1 :green_heart: | mvnsite | 3m 22s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 19m 6s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 1m 39s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 3m 31s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | findbugs | 7m 28s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 11m 20s | hadoop-common in the patch passed. | | -1 :x: | unit | 120m 30s | hadoop-hdfs in the patch passed. | | +1 :green_heart: | asflicense | 0m 52s | The patch does not generate ASF License warnings. | | | | 344m 34s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks | | | hadoop.fs.viewfs.TestViewFsHdfs | | | hadoop.hdfs.TestFileChecksumCompositeCrc | | | hadoop.hdfs.server.datanode.TestBPOfferService | | | hadoop.hdfs.TestSafeModeWithStripedFileWithRandomECPolicy | | | hadoop.hdfs.TestFileChecksum | | | hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics | | | hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier | | | hadoop.hdfs.TestMultipleNNPortQOP | | | hadoop.hdfs.TestStripedFileAppend | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2298/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2298 | | JIRA Issue | HDFS-15532 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 6599a7f3ff4a 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh
[jira] [Work logged] (HDFS-15532) listFiles on root/InternalDir will fail if fallback root has file
[ https://issues.apache.org/jira/browse/HDFS-15532?focusedWorklogId=481996=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-481996 ] ASF GitHub Bot logged work on HDFS-15532: - Author: ASF GitHub Bot Created on: 11/Sep/20 09:57 Start Date: 11/Sep/20 09:57 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #2298: URL: https://github.com/apache/hadoop/pull/2298#issuecomment-690996124 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 30s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | +0 :ok: | mvndep | 3m 34s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 33m 48s | trunk passed | | +1 :green_heart: | compile | 30m 27s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 28m 41s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 4m 25s | trunk passed | | +1 :green_heart: | mvnsite | 4m 28s | trunk passed | | +1 :green_heart: | shadedclient | 29m 14s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 2m 6s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 4m 17s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 4m 24s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 7m 49s | trunk passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 31s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 49s | the patch passed | | +1 :green_heart: | compile | 28m 16s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javac | 28m 16s | the patch passed | | +1 :green_heart: | compile | 23m 51s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | javac | 23m 51s | the patch passed | | -0 :warning: | checkstyle | 3m 33s | root: The patch generated 1 new + 44 unchanged - 0 fixed = 45 total (was 44) | | +1 :green_heart: | mvnsite | 3m 48s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 17m 38s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 2m 31s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 4m 8s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | findbugs | 8m 49s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 14m 50s | hadoop-common in the patch passed. | | -1 :x: | unit | 154m 35s | hadoop-hdfs in the patch passed. | | +1 :green_heart: | asflicense | 1m 51s | The patch does not generate ASF License warnings. | | | | 414m 40s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hdfs.TestFileChecksumCompositeCrc | | | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped | | | hadoop.hdfs.TestFileChecksum | | | hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier | | | hadoop.hdfs.TestMultipleNNPortQOP | | | hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics | | | hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2298/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2298 | | JIRA Issue | HDFS-15532 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux b92e771b2da8 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / 9960c01a25c | | Default Java | Private
[jira] [Work logged] (HDFS-15532) listFiles on root/InternalDir will fail if fallback root has file
[
https://issues.apache.org/jira/browse/HDFS-15532?focusedWorklogId=481864=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-481864
]
ASF GitHub Bot logged work on HDFS-15532:
-
Author: ASF GitHub Bot
Created on: 11/Sep/20 03:00
Start Date: 11/Sep/20 03:00
Worklog Time Spent: 10m
Work Description: umamaheswararao opened a new pull request #2298:
URL: https://github.com/apache/hadoop/pull/2298
https://issues.apache.org/jira/browse/HDFS-15532
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 481864)
Remaining Estimate: 0h
Time Spent: 10m
> listFiles on root/InternalDir will fail if fallback root has file
> -
>
> Key: HDFS-15532
> URL: https://issues.apache.org/jira/browse/HDFS-15532
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> listFiles implementation gets the RemoteIterator created in
> InternalViewFSDirFs as the root is an InternalViewFSDir.
> If there is a fallback and a file exist at root level, it would have
> collected when collecting locatedStatuses.
> When its iterating over to that fallbacks file from RemoteIterator (which
> was returned from InternalViewFSDirFs ), iterator's next will will call
> getFileBlockLocations if it's a file.
> {code:java}
> @Override
> public LocatedFileStatus next() throws IOException {
> System.out.println(this);
> if (!hasNext()) {
> throw new NoSuchElementException("No more entries in " + f);
> }
> FileStatus result = stats[i++];
> // for files, use getBlockLocations(FileStatus, int, int) to avoid
> // calling getFileStatus(Path) to load the FileStatus again
> BlockLocation[] locs = result.isFile() ?
> getFileBlockLocations(result, 0, result.getLen()) :
> null;
> return new LocatedFileStatus(result, locs);
> }{code}
>
> this getFileBlockLocations will be made on InternalViewFSDirFs, as that
> Iterator created originally from that fs.
> InternalViewFSDirFs#getFileBlockLocations does not handle fallback cases.
> It's always expecting "/", this means it always assuming the dir.
> But with the fallback and returning Iterator from InternalViewFSDirFs, will
> create problems.
> Probably we need to handle fallback case in getFileBlockLocations as well.(
> Fallback only should be the reason for call coming to InternalViewFSDirFs
> with other than "/")
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
