[jira] [Updated] (HADOOP-9984) FileSystem#globStatus and FileSystem#listStatus should resolve symlinks by default
[ https://issues.apache.org/jira/browse/HADOOP-9984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HADOOP-9984: Status: Open (was: Patch Available) > FileSystem#globStatus and FileSystem#listStatus should resolve symlinks by > default > -- > > Key: HADOOP-9984 > URL: https://issues.apache.org/jira/browse/HADOOP-9984 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs >Affects Versions: 2.1.0-beta >Reporter: Colin P. McCabe >Assignee: Colin P. McCabe >Priority: Critical > Labels: BB2015-05-TBR > Attachments: HADOOP-9984.001.patch, HADOOP-9984.003.patch, > HADOOP-9984.005.patch, HADOOP-9984.007.patch, HADOOP-9984.009.patch, > HADOOP-9984.010.patch, HADOOP-9984.011.patch, HADOOP-9984.012.patch, > HADOOP-9984.013.patch, HADOOP-9984.014.patch, HADOOP-9984.015.patch > > > During the process of adding symlink support to FileSystem, we realized that > many existing HDFS clients would be broken by listStatus and globStatus > returning symlinks. One example is applications that assume that > !FileStatus#isFile implies that the inode is a directory. As we discussed in > HADOOP-9972 and HADOOP-9912, we should default these APIs to returning > resolved paths. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-9984) FileSystem#globStatus and FileSystem#listStatus should resolve symlinks by default
[ https://issues.apache.org/jira/browse/HADOOP-9984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated HADOOP-9984: Target Version/s: (was: 2.8.0) Not much going on here for a long time, dropping from 2.8.0. Not putting any target-version either anymore, let's target this depending on when there is patch activity. > FileSystem#globStatus and FileSystem#listStatus should resolve symlinks by > default > -- > > Key: HADOOP-9984 > URL: https://issues.apache.org/jira/browse/HADOOP-9984 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs >Affects Versions: 2.1.0-beta >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe >Priority: Critical > Labels: BB2015-05-TBR > Attachments: HADOOP-9984.001.patch, HADOOP-9984.003.patch, > HADOOP-9984.005.patch, HADOOP-9984.007.patch, HADOOP-9984.009.patch, > HADOOP-9984.010.patch, HADOOP-9984.011.patch, HADOOP-9984.012.patch, > HADOOP-9984.013.patch, HADOOP-9984.014.patch, HADOOP-9984.015.patch > > > During the process of adding symlink support to FileSystem, we realized that > many existing HDFS clients would be broken by listStatus and globStatus > returning symlinks. One example is applications that assume that > !FileStatus#isFile implies that the inode is a directory. As we discussed in > HADOOP-9972 and HADOOP-9912, we should default these APIs to returning > resolved paths. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-9984) FileSystem#globStatus and FileSystem#listStatus should resolve symlinks by default
[ https://issues.apache.org/jira/browse/HADOOP-9984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-9984: - Labels: BB2015-05-TBR (was: ) FileSystem#globStatus and FileSystem#listStatus should resolve symlinks by default -- Key: HADOOP-9984 URL: https://issues.apache.org/jira/browse/HADOOP-9984 Project: Hadoop Common Issue Type: Sub-task Components: fs Affects Versions: 2.1.0-beta Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Critical Labels: BB2015-05-TBR Attachments: HADOOP-9984.001.patch, HADOOP-9984.003.patch, HADOOP-9984.005.patch, HADOOP-9984.007.patch, HADOOP-9984.009.patch, HADOOP-9984.010.patch, HADOOP-9984.011.patch, HADOOP-9984.012.patch, HADOOP-9984.013.patch, HADOOP-9984.014.patch, HADOOP-9984.015.patch During the process of adding symlink support to FileSystem, we realized that many existing HDFS clients would be broken by listStatus and globStatus returning symlinks. One example is applications that assume that !FileStatus#isFile implies that the inode is a directory. As we discussed in HADOOP-9972 and HADOOP-9912, we should default these APIs to returning resolved paths. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-9984) FileSystem#globStatus and FileSystem#listStatus should resolve symlinks by default
[ https://issues.apache.org/jira/browse/HADOOP-9984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HADOOP-9984: Target Version/s: 2.4.0 (was: 2.3.0) FileSystem#globStatus and FileSystem#listStatus should resolve symlinks by default -- Key: HADOOP-9984 URL: https://issues.apache.org/jira/browse/HADOOP-9984 Project: Hadoop Common Issue Type: Sub-task Components: fs Affects Versions: 2.1.0-beta Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Blocker Attachments: HADOOP-9984.001.patch, HADOOP-9984.003.patch, HADOOP-9984.005.patch, HADOOP-9984.007.patch, HADOOP-9984.009.patch, HADOOP-9984.010.patch, HADOOP-9984.011.patch, HADOOP-9984.012.patch, HADOOP-9984.013.patch, HADOOP-9984.014.patch, HADOOP-9984.015.patch During the process of adding symlink support to FileSystem, we realized that many existing HDFS clients would be broken by listStatus and globStatus returning symlinks. One example is applications that assume that !FileStatus#isFile implies that the inode is a directory. As we discussed in HADOOP-9972 and HADOOP-9912, we should default these APIs to returning resolved paths. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HADOOP-9984) FileSystem#globStatus and FileSystem#listStatus should resolve symlinks by default
[ https://issues.apache.org/jira/browse/HADOOP-9984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HADOOP-9984: Priority: Critical (was: Blocker) FileSystem#globStatus and FileSystem#listStatus should resolve symlinks by default -- Key: HADOOP-9984 URL: https://issues.apache.org/jira/browse/HADOOP-9984 Project: Hadoop Common Issue Type: Sub-task Components: fs Affects Versions: 2.1.0-beta Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Critical Attachments: HADOOP-9984.001.patch, HADOOP-9984.003.patch, HADOOP-9984.005.patch, HADOOP-9984.007.patch, HADOOP-9984.009.patch, HADOOP-9984.010.patch, HADOOP-9984.011.patch, HADOOP-9984.012.patch, HADOOP-9984.013.patch, HADOOP-9984.014.patch, HADOOP-9984.015.patch During the process of adding symlink support to FileSystem, we realized that many existing HDFS clients would be broken by listStatus and globStatus returning symlinks. One example is applications that assume that !FileStatus#isFile implies that the inode is a directory. As we discussed in HADOOP-9972 and HADOOP-9912, we should default these APIs to returning resolved paths. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HADOOP-9984) FileSystem#globStatus and FileSystem#listStatus should resolve symlinks by default
[ https://issues.apache.org/jira/browse/HADOOP-9984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated HADOOP-9984: --- Target Version/s: 2.3.0 (was: ) FileSystem#globStatus and FileSystem#listStatus should resolve symlinks by default -- Key: HADOOP-9984 URL: https://issues.apache.org/jira/browse/HADOOP-9984 Project: Hadoop Common Issue Type: Sub-task Components: fs Affects Versions: 2.1.0-beta Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Blocker Attachments: HADOOP-9984.001.patch, HADOOP-9984.003.patch, HADOOP-9984.005.patch, HADOOP-9984.007.patch, HADOOP-9984.009.patch, HADOOP-9984.010.patch, HADOOP-9984.011.patch, HADOOP-9984.012.patch, HADOOP-9984.013.patch, HADOOP-9984.014.patch, HADOOP-9984.015.patch During the process of adding symlink support to FileSystem, we realized that many existing HDFS clients would be broken by listStatus and globStatus returning symlinks. One example is applications that assume that !FileStatus#isFile implies that the inode is a directory. As we discussed in HADOOP-9972 and HADOOP-9912, we should default these APIs to returning resolved paths. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HADOOP-9984) FileSystem#globStatus and FileSystem#listStatus should resolve symlinks by default
[ https://issues.apache.org/jira/browse/HADOOP-9984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sanjay Radia updated HADOOP-9984: - Target Version/s: 2.3.0 (was: 2.1.2-beta) Making this a blocker on 2.3 and removing the blocker on 2.1.2-beta since Hadoop-10020 (disabling symlinks) has been committed to branch-2.1-beta. FileSystem#globStatus and FileSystem#listStatus should resolve symlinks by default -- Key: HADOOP-9984 URL: https://issues.apache.org/jira/browse/HADOOP-9984 Project: Hadoop Common Issue Type: Sub-task Components: fs Affects Versions: 2.1.0-beta Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Blocker Attachments: HADOOP-9984.001.patch, HADOOP-9984.003.patch, HADOOP-9984.005.patch, HADOOP-9984.007.patch, HADOOP-9984.009.patch, HADOOP-9984.010.patch, HADOOP-9984.011.patch, HADOOP-9984.012.patch, HADOOP-9984.013.patch, HADOOP-9984.014.patch, HADOOP-9984.015.patch During the process of adding symlink support to FileSystem, we realized that many existing HDFS clients would be broken by listStatus and globStatus returning symlinks. One example is applications that assume that !FileStatus#isFile implies that the inode is a directory. As we discussed in HADOOP-9972 and HADOOP-9912, we should default these APIs to returning resolved paths. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HADOOP-9984) FileSystem#globStatus and FileSystem#listStatus should resolve symlinks by default
[
https://issues.apache.org/jira/browse/HADOOP-9984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HADOOP-9984:
--
Attachment: HADOOP-9984.015.patch
Here is patch #15 adding a change to {{ViewFileSystem}} so that it delegates to
the correct underlying file system for {{listLinkStatus}}.
Reminder: I have only touched the {{FileSystem}} hierarchy in my patches, not
{{AbstractFileSystem}}. That means that if you've already reviewed Colin's
patch #12, then you don't need to re-review the {{AbstractFileSystem}}
hierarchy in this version.
FileSystem#globStatus and FileSystem#listStatus should resolve symlinks by
default
--
Key: HADOOP-9984
URL: https://issues.apache.org/jira/browse/HADOOP-9984
Project: Hadoop Common
Issue Type: Sub-task
Components: fs
Affects Versions: 2.1.0-beta
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Blocker
Attachments: HADOOP-9984.001.patch, HADOOP-9984.003.patch,
HADOOP-9984.005.patch, HADOOP-9984.007.patch, HADOOP-9984.009.patch,
HADOOP-9984.010.patch, HADOOP-9984.011.patch, HADOOP-9984.012.patch,
HADOOP-9984.013.patch, HADOOP-9984.014.patch, HADOOP-9984.015.patch
During the process of adding symlink support to FileSystem, we realized that
many existing HDFS clients would be broken by listStatus and globStatus
returning symlinks. One example is applications that assume that
!FileStatus#isFile implies that the inode is a directory. As we discussed in
HADOOP-9972 and HADOOP-9912, we should default these APIs to returning
resolved paths.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HADOOP-9984) FileSystem#globStatus and FileSystem#listStatus should resolve symlinks by default
[ https://issues.apache.org/jira/browse/HADOOP-9984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HADOOP-9984: - Issue Type: Sub-task (was: Bug) Parent: HADOOP-10019 FileSystem#globStatus and FileSystem#listStatus should resolve symlinks by default -- Key: HADOOP-9984 URL: https://issues.apache.org/jira/browse/HADOOP-9984 Project: Hadoop Common Issue Type: Sub-task Components: fs Affects Versions: 2.1.0-beta Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Blocker Attachments: HADOOP-9984.001.patch, HADOOP-9984.003.patch, HADOOP-9984.005.patch, HADOOP-9984.007.patch, HADOOP-9984.009.patch, HADOOP-9984.010.patch, HADOOP-9984.011.patch, HADOOP-9984.012.patch, HADOOP-9984.013.patch, HADOOP-9984.014.patch During the process of adding symlink support to FileSystem, we realized that many existing HDFS clients would be broken by listStatus and globStatus returning symlinks. One example is applications that assume that !FileStatus#isFile implies that the inode is a directory. As we discussed in HADOOP-9972 and HADOOP-9912, we should default these APIs to returning resolved paths. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HADOOP-9984) FileSystem#globStatus and FileSystem#listStatus should resolve symlinks by default
[
https://issues.apache.org/jira/browse/HADOOP-9984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HADOOP-9984:
--
Attachment: HADOOP-9984.013.patch
Colin, what do you think of this new patch? This implements an idea suggested
by [~sureshms]. The base class still has an abstract {{listStatus}} that
subclasses need to implement. We add {{listLinkStatus}}, but it's not
abstract. If the underlying filesystem does not support symlinks, then it
overrides {{listStatus}} as appropriate, and inherits the default
implementation of {{listLinkStatus}}, which just delegates to {{listStatus}}.
If the underlying file system does support symlinks, then it overrides both
{{listLinkStatus}} and {{listStatus}}. To limit code duplication across the
subclasses, we provide a protected helper method in the base,
{{listStatusAndResolveSymlinks}}. File systems that support symlinks can call
up to this for an easy one-line implementation of {{listStatus}}.
This would be backwards-compatible. As a small side benefit, filesystems that
don't support symlinks wouldn't need to get involved with any *Link methods.
What do you think of this as a compromise? It helps control some of the bad
consequences discussed earlier.
FileSystem#globStatus and FileSystem#listStatus should resolve symlinks by
default
--
Key: HADOOP-9984
URL: https://issues.apache.org/jira/browse/HADOOP-9984
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 2.1.0-beta
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Blocker
Attachments: HADOOP-9984.001.patch, HADOOP-9984.003.patch,
HADOOP-9984.005.patch, HADOOP-9984.007.patch, HADOOP-9984.009.patch,
HADOOP-9984.010.patch, HADOOP-9984.011.patch, HADOOP-9984.012.patch,
HADOOP-9984.013.patch
During the process of adding symlink support to FileSystem, we realized that
many existing HDFS clients would be broken by listStatus and globStatus
returning symlinks. One example is applications that assume that
!FileStatus#isFile implies that the inode is a directory. As we discussed in
HADOOP-9972 and HADOOP-9912, we should default these APIs to returning
resolved paths.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HADOOP-9984) FileSystem#globStatus and FileSystem#listStatus should resolve symlinks by default
[
https://issues.apache.org/jira/browse/HADOOP-9984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HADOOP-9984:
--
Attachment: HADOOP-9984.014.patch
Test failures were caused by missing overrides in {{ChecksumFileSystem}} and
{{ChRootedFileSystem}}. Uploading new patch to fix that.
FileSystem#globStatus and FileSystem#listStatus should resolve symlinks by
default
--
Key: HADOOP-9984
URL: https://issues.apache.org/jira/browse/HADOOP-9984
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 2.1.0-beta
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Blocker
Attachments: HADOOP-9984.001.patch, HADOOP-9984.003.patch,
HADOOP-9984.005.patch, HADOOP-9984.007.patch, HADOOP-9984.009.patch,
HADOOP-9984.010.patch, HADOOP-9984.011.patch, HADOOP-9984.012.patch,
HADOOP-9984.013.patch, HADOOP-9984.014.patch
During the process of adding symlink support to FileSystem, we realized that
many existing HDFS clients would be broken by listStatus and globStatus
returning symlinks. One example is applications that assume that
!FileStatus#isFile implies that the inode is a directory. As we discussed in
HADOOP-9972 and HADOOP-9912, we should default these APIs to returning
resolved paths.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HADOOP-9984) FileSystem#globStatus and FileSystem#listStatus should resolve symlinks by default
[ https://issues.apache.org/jira/browse/HADOOP-9984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HADOOP-9984: - Attachment: HADOOP-9984.011.patch FileSystem#globStatus and FileSystem#listStatus should resolve symlinks by default -- Key: HADOOP-9984 URL: https://issues.apache.org/jira/browse/HADOOP-9984 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 2.1.0-beta Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Blocker Attachments: HADOOP-9984.001.patch, HADOOP-9984.003.patch, HADOOP-9984.005.patch, HADOOP-9984.007.patch, HADOOP-9984.009.patch, HADOOP-9984.010.patch, HADOOP-9984.011.patch, HADOOP-9984.011.patch During the process of adding symlink support to FileSystem, we realized that many existing HDFS clients would be broken by listStatus and globStatus returning symlinks. One example is applications that assume that !FileStatus#isFile implies that the inode is a directory. As we discussed in HADOOP-9972 and HADOOP-9912, we should default these APIs to returning resolved paths. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HADOOP-9984) FileSystem#globStatus and FileSystem#listStatus should resolve symlinks by default
[ https://issues.apache.org/jira/browse/HADOOP-9984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HADOOP-9984: - Attachment: (was: HADOOP-9984.011.patch) FileSystem#globStatus and FileSystem#listStatus should resolve symlinks by default -- Key: HADOOP-9984 URL: https://issues.apache.org/jira/browse/HADOOP-9984 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 2.1.0-beta Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Blocker Attachments: HADOOP-9984.001.patch, HADOOP-9984.003.patch, HADOOP-9984.005.patch, HADOOP-9984.007.patch, HADOOP-9984.009.patch, HADOOP-9984.010.patch, HADOOP-9984.011.patch, HADOOP-9984.012.patch During the process of adding symlink support to FileSystem, we realized that many existing HDFS clients would be broken by listStatus and globStatus returning symlinks. One example is applications that assume that !FileStatus#isFile implies that the inode is a directory. As we discussed in HADOOP-9972 and HADOOP-9912, we should default these APIs to returning resolved paths. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HADOOP-9984) FileSystem#globStatus and FileSystem#listStatus should resolve symlinks by default
[ https://issues.apache.org/jira/browse/HADOOP-9984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HADOOP-9984: - Attachment: HADOOP-9984.012.patch FileSystem#globStatus and FileSystem#listStatus should resolve symlinks by default -- Key: HADOOP-9984 URL: https://issues.apache.org/jira/browse/HADOOP-9984 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 2.1.0-beta Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Blocker Attachments: HADOOP-9984.001.patch, HADOOP-9984.003.patch, HADOOP-9984.005.patch, HADOOP-9984.007.patch, HADOOP-9984.009.patch, HADOOP-9984.010.patch, HADOOP-9984.011.patch, HADOOP-9984.012.patch During the process of adding symlink support to FileSystem, we realized that many existing HDFS clients would be broken by listStatus and globStatus returning symlinks. One example is applications that assume that !FileStatus#isFile implies that the inode is a directory. As we discussed in HADOOP-9972 and HADOOP-9912, we should default these APIs to returning resolved paths. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HADOOP-9984) FileSystem#globStatus and FileSystem#listStatus should resolve symlinks by default
[ https://issues.apache.org/jira/browse/HADOOP-9984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HADOOP-9984: - Attachment: HADOOP-9984.009.patch FileSystem#globStatus and FileSystem#listStatus should resolve symlinks by default -- Key: HADOOP-9984 URL: https://issues.apache.org/jira/browse/HADOOP-9984 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 2.1.0-beta Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Blocker Attachments: HADOOP-9984.001.patch, HADOOP-9984.003.patch, HADOOP-9984.005.patch, HADOOP-9984.007.patch, HADOOP-9984.009.patch During the process of adding symlink support to FileSystem, we realized that many existing HDFS clients would be broken by listStatus and globStatus returning symlinks. One example is applications that assume that !FileStatus#isFile implies that the inode is a directory. As we discussed in HADOOP-9972 and HADOOP-9912, we should default these APIs to returning resolved paths. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HADOOP-9984) FileSystem#globStatus and FileSystem#listStatus should resolve symlinks by default
[ https://issues.apache.org/jira/browse/HADOOP-9984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HADOOP-9984: - Attachment: HADOOP-9984.010.patch Tested the stat on FreeBSD, and that %n should be a %N there. Fixed. FileSystem#globStatus and FileSystem#listStatus should resolve symlinks by default -- Key: HADOOP-9984 URL: https://issues.apache.org/jira/browse/HADOOP-9984 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 2.1.0-beta Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Blocker Attachments: HADOOP-9984.001.patch, HADOOP-9984.003.patch, HADOOP-9984.005.patch, HADOOP-9984.007.patch, HADOOP-9984.009.patch, HADOOP-9984.010.patch During the process of adding symlink support to FileSystem, we realized that many existing HDFS clients would be broken by listStatus and globStatus returning symlinks. One example is applications that assume that !FileStatus#isFile implies that the inode is a directory. As we discussed in HADOOP-9972 and HADOOP-9912, we should default these APIs to returning resolved paths. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HADOOP-9984) FileSystem#globStatus and FileSystem#listStatus should resolve symlinks by default
[
https://issues.apache.org/jira/browse/HADOOP-9984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HADOOP-9984:
--
Attachment: HADOOP-9984.011.patch
Hi, [~cmccabe]. I was still seeing failures on Mac/FreeBSD. The problem was
due to hard-coding of {{`link'}} in the stat escape sequence, where the
subsequent logic depends on exactly 7 characters length difference compared to
the source path.
I'm attaching version 11 of the patch with a small change to make this insert
the value of the source {{Path#toString}}. Tests are passing on Mac for me
with this version of the patch. For reference, here is the incremental diff of
version 10 to version 11:
{code}
[chris@Chriss-MacBook-Pro:ttys004] hadoop-common
(git:trunk)
diff ~/patch/HADOOP-9984.010.patch ~/patch/HADOOP-9984.011.patch
1098c1098
index c2ec63c..580a509 100644
---
index c2ec63c..df9460c 100644
1101c1101
@@ -99,10 +99,10 @@ FileStatus getFileStatusForTesting() {
---
@@ -99,10 +99,13 @@ FileStatus getFileStatusForTesting() {
1106c1106,1108
+ stat, derefFlag + c, %s,%F,%Y,%X,%a,%U,%G,%n,%N,
path.toString() };
---
+ stat, derefFlag + c,
+ %s,%F,%Y,%X,%a,%U,%G,%n,%N,
+ path.toString() };
1110c1112,1113
+ stat, derefFlag + f, %z,%HT,%m,%a,%Op,%Su,%Sg,%N,`link' -
`%Y',
---
+ stat, derefFlag + f,
+ %z,%HT,%m,%a,%Op,%Su,%Sg,%N,` + path.toString() + ' - `%Y',
1114c1117
@@ -145,21 +145,32 @@ protected void parseExecResult(BufferedReader lines)
throws IOException {
---
@@ -145,21 +148,32 @@ protected void parseExecResult(BufferedReader lines)
throws IOException {
{code}
FileSystem#globStatus and FileSystem#listStatus should resolve symlinks by
default
--
Key: HADOOP-9984
URL: https://issues.apache.org/jira/browse/HADOOP-9984
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 2.1.0-beta
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Blocker
Attachments: HADOOP-9984.001.patch, HADOOP-9984.003.patch,
HADOOP-9984.005.patch, HADOOP-9984.007.patch, HADOOP-9984.009.patch,
HADOOP-9984.010.patch, HADOOP-9984.011.patch
During the process of adding symlink support to FileSystem, we realized that
many existing HDFS clients would be broken by listStatus and globStatus
returning symlinks. One example is applications that assume that
!FileStatus#isFile implies that the inode is a directory. As we discussed in
HADOOP-9972 and HADOOP-9912, we should default these APIs to returning
resolved paths.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HADOOP-9984) FileSystem#globStatus and FileSystem#listStatus should resolve symlinks by default
[ https://issues.apache.org/jira/browse/HADOOP-9984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HADOOP-9984: - Attachment: HADOOP-9984.007.patch FileSystem#globStatus and FileSystem#listStatus should resolve symlinks by default -- Key: HADOOP-9984 URL: https://issues.apache.org/jira/browse/HADOOP-9984 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 2.1.0-beta Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Blocker Attachments: HADOOP-9984.001.patch, HADOOP-9984.003.patch, HADOOP-9984.005.patch, HADOOP-9984.007.patch During the process of adding symlink support to FileSystem, we realized that many existing HDFS clients would be broken by listStatus and globStatus returning symlinks. One example is applications that assume that !FileStatus#isFile implies that the inode is a directory. As we discussed in HADOOP-9972 and HADOOP-9912, we should default these APIs to returning resolved paths. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9984) FileSystem#globStatus and FileSystem#listStatus should resolve symlinks by default
[ https://issues.apache.org/jira/browse/HADOOP-9984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HADOOP-9984: - Attachment: HADOOP-9984.005.patch FileSystem#globStatus and FileSystem#listStatus should resolve symlinks by default -- Key: HADOOP-9984 URL: https://issues.apache.org/jira/browse/HADOOP-9984 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 2.1.0-beta Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Blocker Attachments: HADOOP-9984.001.patch, HADOOP-9984.003.patch, HADOOP-9984.005.patch During the process of adding symlink support to FileSystem, we realized that many existing HDFS clients would be broken by listStatus and globStatus returning symlinks. One example is applications that assume that !FileStatus#isFile implies that the inode is a directory. As we discussed in HADOOP-9972 and HADOOP-9912, we should default these APIs to returning resolved paths. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9984) FileSystem#globStatus and FileSystem#listStatus should resolve symlinks by default
[ https://issues.apache.org/jira/browse/HADOOP-9984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HADOOP-9984: - Summary: FileSystem#globStatus and FileSystem#listStatus should resolve symlinks by default (was: FileSystem#listStatus should resolve symlinks by default) FileSystem#globStatus and FileSystem#listStatus should resolve symlinks by default -- Key: HADOOP-9984 URL: https://issues.apache.org/jira/browse/HADOOP-9984 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 2.1.0-beta Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Blocker Attachments: HADOOP-9984.001.patch, HADOOP-9984.003.patch During the process of adding symlink support to FileSystem, we realized that many existing HDFS clients would be broken by listStatus and globStatus returning symlinks. One example is applications that assume that !FileStatus#isFile implies that the inode is a directory. As we discussed in HADOOP-9972 and HADOOP-9912, we should default these APIs to returning resolved paths. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9984) FileSystem#globStatus and FileSystem#listStatus should resolve symlinks by default
[ https://issues.apache.org/jira/browse/HADOOP-9984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HADOOP-9984: - Component/s: fs Priority: Blocker (was: Major) Target Version/s: 2.1.0-beta FileSystem#globStatus and FileSystem#listStatus should resolve symlinks by default -- Key: HADOOP-9984 URL: https://issues.apache.org/jira/browse/HADOOP-9984 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 2.1.0-beta Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Blocker During the process of adding symlink support to FileSystem, we realized that many existing HDFS clients would be broken by listStatus and globStatus returning symlinks. One example is applications that assume that !FileStatus#isFile implies that the inode is a directory. As we discussed in HADOOP-9972 and HADOOP-9912, we should default these APIs to returning resolved paths. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9984) FileSystem#globStatus and FileSystem#listStatus should resolve symlinks by default
[ https://issues.apache.org/jira/browse/HADOOP-9984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HADOOP-9984: - Attachment: HADOOP-9984.001.patch Posting a preliminary version. I expect the full jenkins run will identify a few tests that need to be updated here. FileSystem#globStatus and FileSystem#listStatus should resolve symlinks by default -- Key: HADOOP-9984 URL: https://issues.apache.org/jira/browse/HADOOP-9984 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 2.1.0-beta Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Blocker Attachments: HADOOP-9984.001.patch During the process of adding symlink support to FileSystem, we realized that many existing HDFS clients would be broken by listStatus and globStatus returning symlinks. One example is applications that assume that !FileStatus#isFile implies that the inode is a directory. As we discussed in HADOOP-9972 and HADOOP-9912, we should default these APIs to returning resolved paths. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
