[
https://issues.apache.org/jira/browse/HDFS-9260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14963610#comment-14963610
]
Staffan Friberg commented on HDFS-9260:
---------------------------------------
Hi Walter
1)
I think this is part could reduce the need to go off heap. If we still see
scalability issues and need to put the block map and blockinfo off heap, this
could potenially serve as an idea on how to structure the data of heap, since
even if the data is off heap continuously updating reference will be costly
since we will invalidate the CPU cache. Potentially a version of the TreeSet
holding primitives (blockinfo address) could be used for fast iteration, but
need to think a bit further about.
2)
Interesting idea, I think the key point would be that you could do quick lookup
directly on the serialized data so you don't need to instantiate the whole map
since it might be rather large. Not sure if this is easily doable with ProtoBuf
and still keeping the message as compact as possible?
> Improve performance and GC friendliness of startup and FBRs
> -----------------------------------------------------------
>
> Key: HDFS-9260
> URL: https://issues.apache.org/jira/browse/HDFS-9260
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode, namenode, performance
> Affects Versions: 2.7.1
> Reporter: Staffan Friberg
> Assignee: Staffan Friberg
> Attachments: HDFS Block and Replica Management 20151013.pdf,
> HDFS-7435.001.patch, HDFS-7435.002.patch, HDFS-7435.003.patch
>
>
> This patch changes the datastructures used for BlockInfos and Replicas to
> keep them sorted. This allows faster and more GC friendly handling of full
> block reports.
> Would like to hear peoples feedback on this change and also some help
> investigating/understanding a few outstanding issues if we are interested in
> moving forward with this.
> There seems to be some timing issues I hit when testing the patch, not sure
> if it is a bug in the patch or something else (most likely the earlier)...
> Tests that fail for me:
> The issues seems to be that the blocks are not on any storage, so no
> replication can occur causing the tests to fail in different ways.
> TestDecomission.testDecommision
> If I add a little sleep after the cleanup/delete things seem to work
> TestDFSStripedOutputStreamWithFailure
> A couple of tests fails in this class.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
