[
https://issues.apache.org/jira/browse/HDFS-13616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16491009#comment-16491009
]
Todd Lipcon commented on HDFS-13616:
------------------------------------
bq. Why not just RemoteIterator<FileStatus>?
In an earlier prototype of this patch, Andrew had it this way, but I requested
the change to the new API, which groups together all of the results for a given
parent directory. This makes it much easier to associate back the returned
FileStatus to the directories being requested. Certainly it would be possible
to iterate over each FileStatus, get the Path, and then get the parent for each
one to prefix match back, but that wastes a bit of CPU in URI parsing, etc.
One question about the API for Andrew, actually: if I pass a file (instead of a
directory), will I get back a standalone PartialListing that only includes the
FileStatus for that file? If so, is that indicated differently in any way? Will
'batch.getParent' end up returning the file or its parent? It's probably worth
documenting the semantics there.
bq. Does that even fit the use case? (I'm guessing no, only some of the
partition dirs in the parent need listing–but we need to justify any new
FileSystem surface area).
Right, it doesn't fit the use case. A table might have thousands of partitions,
but with a query like 'select * where time = 123' it's likely to be able to
prune most of them and only need to list the files in the remaining few.
Additionally, it's permitted for users to specify custom partition locations,
so a given table may have partitions located in different spots on the file
system, even though the common case is that they all share a parent directory.
> Batch listing of multiple directories
> -------------------------------------
>
> Key: HDFS-13616
> URL: https://issues.apache.org/jira/browse/HDFS-13616
> Project: Hadoop HDFS
> Issue Type: New Feature
> Affects Versions: 3.2.0
> Reporter: Andrew Wang
> Assignee: Andrew Wang
> Priority: Major
> Attachments: BenchmarkListFiles.java, HDFS-13616.001.patch,
> HDFS-13616.002.patch
>
>
> One of the dominant workloads for external metadata services is listing of
> partition directories. This can end up being bottlenecked on RTT time when
> partition directories contain a small number of files. This is fairly common,
> since fine-grained partitioning is used for partition pruning by the query
> engines.
> A batched listing API that takes multiple paths amortizes the RTT cost.
> Initial benchmarks show a 10-20x improvement in metadata loading performance.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
