[
https://issues.apache.org/jira/browse/HDFS-11634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966916#comment-15966916
]
Zhe Zhang commented on HDFS-11634:
----------------------------------
Thanks [~shv] for the patch and [~shahrs87] for the review. The idea looks good
to me, it's a good improvement to skip entire storages.
I only have one concern/question:
{code}
for (DatanodeStorageInfo e : storages) {
iterators.add(e.getBlockIterator());
int numBlocks = e.numBlocks();
sumBlocks += numBlocks;
if(sumBlocks <= startBlock) {
index++;
s -= numBlocks;
}
}
{code}
If a storage is skipped, should we still add it to the {{iterators}}?
> Optimize BlockIterator when interating starts in the middle.
> ------------------------------------------------------------
>
> Key: HDFS-11634
> URL: https://issues.apache.org/jira/browse/HDFS-11634
> Project: Hadoop HDFS
> Issue Type: Improvement
> Affects Versions: 2.6.5
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Attachments: HDFS-11634.001.patch, HDFS-11634.002.patch,
> HDFS-11634.003.patch, HDFS-11634.004.patch
>
>
> {{BlockManager.getBlocksWithLocations()}} needs to iterate blocks from a
> randomly selected {{startBlock}} index. It creates an iterator which points
> to the first block and then skips all blocks until {{startBlock}}. It is
> inefficient when DN has multiple storages. Instead of skipping blocks one by
> one we can skip entire storages. Should be more efficient on average.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
