[
https://issues.apache.org/jira/browse/MAPREDUCE-6760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15426143#comment-15426143
]
Steve Loughran commented on MAPREDUCE-6760:
-------------------------------------------
Note that if we find that listFiles isn't completely what we want (it skips
dirs), we may want to extend that FS method, as we can update the object stores
and MR classes in sync. Things like Hive wouldn't need to change
> LocatedFileStatusFetcher to use listFiles(recursive)
> ----------------------------------------------------
>
> Key: MAPREDUCE-6760
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6760
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 2.8.0
> Reporter: Steve Loughran
>
> {{LocatedFileStatusFetcher }} does parallelized path listing, but it does
> make recursive calls on every subdir.
> If we could switch it to use {{FileSystem.listFiles(recursive)}}, object
> stores that have high-performance implementations of that operation would see
> significant speedup.
> HADOOP-13208 implements that for S3A; Azure, swift &c can do the same.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
