[
https://issues.apache.org/jira/browse/HDFS-13616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16497016#comment-16497016
]
Andrew Wang commented on HDFS-13616:
------------------------------------
Would we be okay with making this a DistributedFileSystem-only change, with no
changes to FileSystem? Right now non-DFS will just throw
UnsupportedOperationException anyway, so it's specific to HDFS as it is.
The batch feed API seems reasonable but it does add a lot of scope. I'm not
sure how to apply it to stream-oriented operations like reading and writing
files, so it may only be usable for metadata operations. So if we're okay with
a limited set of operations initially (say, just listing and delete), then I
could look into it.
> Batch listing of multiple directories
> -------------------------------------
>
> Key: HDFS-13616
> URL: https://issues.apache.org/jira/browse/HDFS-13616
> Project: Hadoop HDFS
> Issue Type: New Feature
> Affects Versions: 3.2.0
> Reporter: Andrew Wang
> Assignee: Andrew Wang
> Priority: Major
> Attachments: BenchmarkListFiles.java, HDFS-13616.001.patch,
> HDFS-13616.002.patch
>
>
> One of the dominant workloads for external metadata services is listing of
> partition directories. This can end up being bottlenecked on RTT time when
> partition directories contain a small number of files. This is fairly common,
> since fine-grained partitioning is used for partition pruning by the query
> engines.
> A batched listing API that takes multiple paths amortizes the RTT cost.
> Initial benchmarks show a 10-20x improvement in metadata loading performance.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
