[
https://issues.apache.org/jira/browse/HDFS-11634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16058203#comment-16058203
]
Zhe Zhang edited comment on HDFS-11634 at 6/21/17 8:51 PM:
-----------------------------------------------------------
On one of our clusters with ~150 million blocks, {{getBlocks}} used to take
40ms. With the change it is taking 9ms, more than *4x* faster! This also leads
to a significant improvement on NameNode RPC performance when Balancer is
running.
was (Author: zhz):
On our cluster with ~150 million blocks, {{getBlocks}} used to take 40ms. With
the change it is taking 9ms, more than *4x* faster! This also leads to a
significant improvement on NameNode RPC performance when Balancer is running.
> Optimize BlockIterator when iterating starts in the middle.
> -----------------------------------------------------------
>
> Key: HDFS-11634
> URL: https://issues.apache.org/jira/browse/HDFS-11634
> Project: Hadoop HDFS
> Issue Type: Improvement
> Affects Versions: 2.6.5
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Fix For: 2.9.0, 2.7.4, 3.0.0-alpha4, 2.8.2
>
> Attachments: HDFS-11634.001.patch, HDFS-11634.002.patch,
> HDFS-11634.003.patch, HDFS-11634.004.patch, HDFS-11634.006.patch,
> HDFS-11634-branch-2.7.006.patch, HDFS-11643.005.patch, Screen Shot 2017-06-21
> at 1.39.21 PM.png
>
>
> {{BlockManager.getBlocksWithLocations()}} needs to iterate blocks from a
> randomly selected {{startBlock}} index. It creates an iterator which points
> to the first block and then skips all blocks until {{startBlock}}. It is
> inefficient when DN has multiple storages. Instead of skipping blocks one by
> one we can skip entire storages. Should be more efficient on average.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
