[
https://issues.apache.org/jira/browse/HADOOP-9981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13773329#comment-13773329
]
Kihwal Lee commented on HADOOP-9981:
------------------------------------
I think the new Globber added in HADOOP-9817 causes listStaus() of parents when
globStatus() is called as in the Hive use case. The large number of files were
actually in the parent directory of the data directory. While fixing stat will
make it go faster, the Globber needs to be fixed as well to avoid unnecessary
stats.
> Listing in RawLocalFileSystem is inefficient
> --------------------------------------------
>
> Key: HADOOP-9981
> URL: https://issues.apache.org/jira/browse/HADOOP-9981
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 2.3.0
> Reporter: Kihwal Lee
> Priority: Critical
>
> After HADOOP-9652, listStatus() or globStatus() calls against a local file
> system directory is very slow. A user was loading data from local file
> system to Hive and it took about 30 seconds. The same operation took less
> than a second pre-HADOOP-9652.
> The input path had many other files beside the input files and strace showed
> that fork & exec of stat against each and every one of them. jstack confirmed
> that this was being done from getNativeFileLinkStatus().
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
