[
https://issues.apache.org/jira/browse/HDFS-2477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13568710#comment-13568710
]
Hudson commented on HDFS-2477:
------------------------------
Integrated in Hadoop-Hdfs-0.23-Build #512 (See
[https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/512/])
Merging r1196676 and r1197801 from trunk to branch-0.23 to fix HDFS-2477
(Revision 1441131)
Result = SUCCESS
kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1441131
Files :
* /hadoop/common/branches/branch-0.23
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo.java
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInfo.java
> Optimize computing the diff between a block report and the namenode state.
> --------------------------------------------------------------------------
>
> Key: HDFS-2477
> URL: https://issues.apache.org/jira/browse/HDFS-2477
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
> Affects Versions: 0.23.0
> Reporter: Tomasz Nykiel
> Assignee: Tomasz Nykiel
> Fix For: 2.0.0-alpha, 0.23.7
>
> Attachments: reportDiff.patch, reportDiff.patch-2,
> reportDiff.patch-3, reportDiff.patch-4, reportDiff.patch-5
>
>
> When a block report is processed at the NN, the BlockManager.reportDiff
> traverses all blocks contained in the report, and for each one block, which
> is also present in the corresponding datanode descriptor, the block is moved
> to the head of the list of the blocks in this datanode descriptor.
> With HDFS-395 the huge majority of the blocks in the report, are also present
> in the datanode descriptor, which means that almost every block in the report
> will have to be moved to the head of the list.
> Currently this operation is performed by DatanodeDescriptor.moveBlockToHead,
> which removes a block from a list and then inserts it. In this process, we
> call findDatanode several times (afair 6 times for each moveBlockToHead
> call). findDatanode is relatively expensive, since it linearly goes through
> the triplets to locate the given datanode.
> With this patch, we do some memoization of findDatanode, so we can reclaim 2
> findDatanode calls. Our experiments show that this can improve the reportDiff
> (which is executed under write lock) by around 15%. Currently with HDFS-395,
> reportDiff is responsible for almost 100% of the block report processing time.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
