[
https://issues.apache.org/jira/browse/HDFS-6658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14372508#comment-14372508
]
Colin Patrick McCabe commented on HDFS-6658:
--------------------------------------------
Daryn, I apologize for not being more responsive on this. I've been dealing
with some burning fires around here and haven't had time to look at it more.
It would be nice if this could help with the goals of HDFS-7836, especially
multi-threading block report processing and getting the heap below 32GB in the
long term. Right now I don't see a path from this patch to there but very
possibly I'm missing something. Let's chat about it sometime next week.
> Namenode memory optimization - Block replicas list
> ---------------------------------------------------
>
> Key: HDFS-6658
> URL: https://issues.apache.org/jira/browse/HDFS-6658
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
> Affects Versions: 2.4.1
> Reporter: Amir Langer
> Assignee: Daryn Sharp
> Attachments: BlockListOptimizationComparison.xlsx, BlocksMap
> redesign.pdf, HDFS-6658.patch, HDFS-6658.patch, HDFS-6658.patch, Namenode
> Memory Optimizations - Block replicas list.docx, New primative indexes.jpg,
> Old triplets.jpg
>
>
> Part of the memory consumed by every BlockInfo object in the Namenode is a
> linked list of block references for every DatanodeStorageInfo (called
> "triplets").
> We propose to change the way we store the list in memory.
> Using primitive integer indexes instead of object references will reduce the
> memory needed for every block replica (when compressed oops is disabled) and
> in our new design the list overhead will be per DatanodeStorageInfo and not
> per block replica.
> see attached design doc. for details and evaluation results.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
