[
https://issues.apache.org/jira/browse/HDFS-6658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daryn Sharp updated HDFS-6658:
------------------------------
Attachment: HDFS-6658.patch
There's likely to be some todo debris, maybe snippets of other work, and some
tests might fail, but this is the fruit of a multi-month effort. I was able to
bench an earlier prototype to have comparable block report processing times
+/-5%. Since then, I've lost sleep ensuring there are adequate precondition
and co-modification checks to prevent the data structures from going off the
rails and scribbling itself to death.
I need to re-benchmark on a large perf cluster to make sure I didn't regress on
performance.
The change looks big, but it's actually a lot of tests, and I feel I need even
more. The pre-existing logic has generally just moved. There's no fundamental
changes.
The main change required to make this primitive array approach work is
requiring the block manager to manage all block related data structures. The
DNDs and DNSIs become "dumb" model objects controlled by the BM.
> Namenode memory optimization - Block replicas list
> ---------------------------------------------------
>
> Key: HDFS-6658
> URL: https://issues.apache.org/jira/browse/HDFS-6658
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
> Affects Versions: 2.4.1
> Reporter: Amir Langer
> Assignee: Daryn Sharp
> Attachments: BlockListOptimizationComparison.xlsx, BlocksMap
> redesign.pdf, HDFS-6658.patch, HDFS-6658.patch, Namenode Memory Optimizations
> - Block replicas list.docx
>
>
> Part of the memory consumed by every BlockInfo object in the Namenode is a
> linked list of block references for every DatanodeStorageInfo (called
> "triplets").
> We propose to change the way we store the list in memory.
> Using primitive integer indexes instead of object references will reduce the
> memory needed for every block replica (when compressed oops is disabled) and
> in our new design the list overhead will be per DatanodeStorageInfo and not
> per block replica.
> see attached design doc. for details and evaluation results.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
