[
https://issues.apache.org/jira/browse/HDFS-13616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16490167#comment-16490167
]
Xiao Chen commented on HDFS-13616:
----------------------------------
Thanks for the work here Andrew, and others for the comments! I had fun reading
through. :)
Some semantic questions:
- We currently FNFE on the first error. Is it possible a partition is deleted
while another thread is listing halfway for Hive/Impala? What's the expected
behavior from them if so? (I'm lacking the knowledge of this so no strong
preference either way, but curious...)
- If caller added some subdirs to srcs, should we list the subdir twice, or
throw, or 'smartly' list everything at most once?
> Batch listing of multiple directories
> -------------------------------------
>
> Key: HDFS-13616
> URL: https://issues.apache.org/jira/browse/HDFS-13616
> Project: Hadoop HDFS
> Issue Type: New Feature
> Affects Versions: 3.2.0
> Reporter: Andrew Wang
> Assignee: Andrew Wang
> Priority: Major
> Attachments: HDFS-13616.001.patch, HDFS-13616.002.patch
>
>
> One of the dominant workloads for external metadata services is listing of
> partition directories. This can end up being bottlenecked on RTT time when
> partition directories contain a small number of files. This is fairly common,
> since fine-grained partitioning is used for partition pruning by the query
> engines.
> A batched listing API that takes multiple paths amortizes the RTT cost.
> Initial benchmarks show a 10-20x improvement in metadata loading performance.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
