[
https://issues.apache.org/jira/browse/MAPREDUCE-1501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12836963#action_12836963
]
Zheng Shao commented on MAPREDUCE-1501:
---------------------------------------
Thanks Dhruba. I missed the part "and other hidden directories". We do call
PathFilter on the sub directories as well (see addInputPathRecursively(...)).
Is that good enough or we want to split the PathFilters for files and the
PathFilters for directories?
> FileInputFormat to support multi-level/recursive directory listing
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-1501
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1501
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Attachments: MAPREDUCE-1501.1.branch-0.20.patch,
> MAPREDUCE-1501.1.trunk.patch
>
>
> As we have seen multiple times in the mailing list, users want to have the
> capability of getting all files out of a multi-level directory structure.
> 4/1/2008:
> http://mail-archives.apache.org/mod_mbox/hadoop-core-user/200804.mbox/%3ce75c02ef0804011433x144813e6x2450da7883de3...@mail.gmail.com%3e
> 2/3/2009:
> http://mail-archives.apache.org/mod_mbox/hadoop-core-user/200902.mbox/%3c7f80089c-3e7f-4330-90ba-6f1c5b0b0...@nist.gov%3e
> 6/2/2009:
> http://mail-archives.apache.org/mod_mbox/hadoop-common-user/200906.mbox/%3c4a258a16.8050...@darose.net%3e
> One solution that our users had is to write a new FileInputFormat, but that
> means all existing FileInputFormat subclasses need to be changed in order to
> support this feature.
> We can easily provide a JobConf option (which defaults to false) to
> {{FileInputFormat.listStatus(...)}} to recursively go into directory
> structure.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
