[
https://issues.apache.org/jira/browse/HDFS-9260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14964712#comment-14964712
]
Walter Su commented on HDFS-9260:
---------------------------------
The biggest part of BlockInfo is the implicit linked lists. If we move the
lists to TreeSet, Does it go against the effort of moving BlockInfo off-heap?
The TreeMap brings more on-heap reference to off-heap BlockInfo. {{BlocksMap}}
is a reference map still need to be off-heap.(HDFS-7846). I think TreeMap need
to be an off-heap set as well. Since set/map have the same numbers of blocks,
and block number grows. But does it worth it?
The point of TreeSet is for fast iteration for processing FBR. As said by
[~cmccabe] in
[here|https://issues.apache.org/jira/browse/HDFS-7846?focusedCommentId=14355701&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14355701],
iterating over all the blocks on a given datanode isn't quite necessary during
FBR. Although iteration still needed in other use-cases.
> Improve performance and GC friendliness of startup and FBRs
> -----------------------------------------------------------
>
> Key: HDFS-9260
> URL: https://issues.apache.org/jira/browse/HDFS-9260
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode, namenode, performance
> Affects Versions: 2.7.1
> Reporter: Staffan Friberg
> Assignee: Staffan Friberg
> Attachments: HDFS Block and Replica Management 20151013.pdf,
> HDFS-7435.001.patch, HDFS-7435.002.patch, HDFS-7435.003.patch
>
>
> This patch changes the datastructures used for BlockInfos and Replicas to
> keep them sorted. This allows faster and more GC friendly handling of full
> block reports.
> Would like to hear peoples feedback on this change and also some help
> investigating/understanding a few outstanding issues if we are interested in
> moving forward with this.
> There seems to be some timing issues I hit when testing the patch, not sure
> if it is a bug in the patch or something else (most likely the earlier)...
> Tests that fail for me:
> The issues seems to be that the blocks are not on any storage, so no
> replication can occur causing the tests to fail in different ways.
> TestDecomission.testDecommision
> If I add a little sleep after the cleanup/delete things seem to work
> TestDFSStripedOutputStreamWithFailure
> A couple of tests fails in this class.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
