[
https://issues.apache.org/jira/browse/HDFS-13616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16489981#comment-16489981
]
Todd Lipcon edited comment on HDFS-13616 at 5/24/18 11:32 PM:
--------------------------------------------------------------
Actually collecting that was easier than I thought. I found a table with 28509
partitions and only 73400 files (5500 of the partitions are even empty). With
the batched approach, average NN CPU consumption is 2.58sec of CPU. With the
5-threaded threadpool approach, it's 5.78sec of CPU (2.24x improvement). For
this table it also reduces the number of round trips enough that the wall-time
of fetching the partitions to Impala went from 15.5sec down to 8.0sec.
In my experience neither type of table is uncommon - we see some tables with
lots of partitions, each of which is large, and some tables with lots of
partitions each containing a very small handful of files. I just grabbed a few
random tables from a customer workload and found both types.The benefit is much
larger for the tables like the latter, but this shouldn't be detrimental for
the former either.
was (Author: tlipcon):
Actually collecting that was easier than I thought. I found a table with 28509
partitions and only 73400 tables (5500 of the partitions are even empty). With
the batched approach, average NN CPU consumption is 2.58sec of CPU. With the
5-threaded threadpool approach, it's 5.78sec of CPU (2.24x improvement). For
this table it also reduces the number of round trips enough that the wall-time
of fetching the partitions to Impala went from 15.5sec down to 8.0sec.
In my experience neither type of table is uncommon - we see some tables with
lots of partitions, each of which is large, and some tables with lots of
partitions each containing a very small handful of files. I just grabbed a few
random tables from a customer workload and found both types.The benefit is much
larger for the tables like the latter, but this shouldn't be detrimental for
the former either.
> Batch listing of multiple directories
> -------------------------------------
>
> Key: HDFS-13616
> URL: https://issues.apache.org/jira/browse/HDFS-13616
> Project: Hadoop HDFS
> Issue Type: New Feature
> Affects Versions: 3.2.0
> Reporter: Andrew Wang
> Assignee: Andrew Wang
> Priority: Major
> Attachments: HDFS-13616.001.patch
>
>
> One of the dominant workloads for external metadata services is listing of
> partition directories. This can end up being bottlenecked on RTT time when
> partition directories contain a small number of files. This is fairly common,
> since fine-grained partitioning is used for partition pruning by the query
> engines.
> A batched listing API that takes multiple paths amortizes the RTT cost.
> Initial benchmarks show a 10-20x improvement in metadata loading performance.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
