[
https://issues.apache.org/jira/browse/HDFS-2477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13567960#comment-13567960
]
Kihwal Lee commented on HDFS-2477:
----------------------------------
Merged this to branch-0.23. The 0.23.3 became branch-2 and the current
branch-0.23 was missing this.
> Optimize computing the diff between a block report and the namenode state.
> --------------------------------------------------------------------------
>
> Key: HDFS-2477
> URL: https://issues.apache.org/jira/browse/HDFS-2477
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
> Affects Versions: 0.23.0
> Reporter: Tomasz Nykiel
> Assignee: Tomasz Nykiel
> Fix For: 2.0.0-alpha
>
> Attachments: reportDiff.patch, reportDiff.patch-2,
> reportDiff.patch-3, reportDiff.patch-4, reportDiff.patch-5
>
>
> When a block report is processed at the NN, the BlockManager.reportDiff
> traverses all blocks contained in the report, and for each one block, which
> is also present in the corresponding datanode descriptor, the block is moved
> to the head of the list of the blocks in this datanode descriptor.
> With HDFS-395 the huge majority of the blocks in the report, are also present
> in the datanode descriptor, which means that almost every block in the report
> will have to be moved to the head of the list.
> Currently this operation is performed by DatanodeDescriptor.moveBlockToHead,
> which removes a block from a list and then inserts it. In this process, we
> call findDatanode several times (afair 6 times for each moveBlockToHead
> call). findDatanode is relatively expensive, since it linearly goes through
> the triplets to locate the given datanode.
> With this patch, we do some memoization of findDatanode, so we can reclaim 2
> findDatanode calls. Our experiments show that this can improve the reportDiff
> (which is executed under write lock) by around 15%. Currently with HDFS-395,
> reportDiff is responsible for almost 100% of the block report processing time.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
