[
https://issues.apache.org/jira/browse/HADOOP-9984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799499#comment-13799499
]
Sanjay Radia commented on HADOOP-9984:
--------------------------------------
*Background:*
Applications often call listStatus and then call fileStatus.isDir() on each of
the retuned children to decide if a node is a dir or a file. Such code would
potentially break if any of the children are symlinks. This jira proposed that
listStatus should follow any child symlinks and return a resolved list of
children. Note symlinks that occur in the pathname passed to listStatus are
always transparently followed and are not an issue. Also note that when
symlinks was introduced, isDir() was deprecated and isDirectory(), isFile(),
iSymlink() were added.
*Compare with Posix:*
Posix has separate readDir and stat/lstat. While readDir does not return the
full status of each child, it does return the file type in the struct-dirent
(i.e. regular file, dir, symlink etc).
*Issue with following child symlinks*
This lira's proposed solution (follow the child symlinks) has an issue. Comment
[by
daryn|https://issues.apache.org/jira/browse/HADOOP-9984?focusedCommentId=13786431&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13786431]
and
[Oct9th|https://issues.apache.org/jira/browse/HADOOP-9984?focusedCommentId=13790972&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13790972]
in this jira shows potential problems with following child symlinks - the
most egregious being the duplicate entry.
*New Proposed Solution*
listStatus should NOT follow child symlinks. Fix all internal utilities, hive,
pig, map reduce, yarn, etc to not use isDir() and understand that a directory
may contain symlinks.
We have two choices for isDir() (which, btw, has already been deprecated)
a) isDir() returns the file type of child without following the symlink (this
is the code in trunk)
b) isDir() returns the file type of child after following the symlink. (
unless the link is dangling).
My own preference is (a). The argument in favor of (b) is that it would provide
greater compatibility.
I think regardless of which choice we pick we will break some apps; in that
case I rather pick the cleaner solution, (a).
> FileSystem#globStatus and FileSystem#listStatus should resolve symlinks by
> default
> ----------------------------------------------------------------------------------
>
> Key: HADOOP-9984
> URL: https://issues.apache.org/jira/browse/HADOOP-9984
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
> Affects Versions: 2.1.0-beta
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Blocker
> Attachments: HADOOP-9984.001.patch, HADOOP-9984.003.patch,
> HADOOP-9984.005.patch, HADOOP-9984.007.patch, HADOOP-9984.009.patch,
> HADOOP-9984.010.patch, HADOOP-9984.011.patch, HADOOP-9984.012.patch,
> HADOOP-9984.013.patch, HADOOP-9984.014.patch, HADOOP-9984.015.patch
>
>
> During the process of adding symlink support to FileSystem, we realized that
> many existing HDFS clients would be broken by listStatus and globStatus
> returning symlinks. One example is applications that assume that
> !FileStatus#isFile implies that the inode is a directory. As we discussed in
> HADOOP-9972 and HADOOP-9912, we should default these APIs to returning
> resolved paths.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
