[
https://issues.apache.org/jira/browse/HADOOP-803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12466764
]
Raghu Angadi commented on HADOOP-803:
-------------------------------------
On a small cluster with 3035 blocks verified blocks references work as expected
using netbeans profiler:
Number of Block objects :
before the patch : 5*3035 ( 3 replicas, 1 in blockMap, 1 in Inode/File )
after the patch : 2 * 3035 ( in blockMap and inode/File).
If the blocks were created after Namenode is started, one in inode will share
the object with blockMap. When name nodes starts up, it initially creates all
the blocks in Indode while reading the image file.. does not seem easy to share
that reference.
TreeMap.Entry objects also reduced from 20k to around 11.5 k. Due to changing
containingNodes to ArrayList instead of TreeMap.
TreeMap.Entry and Block used to take max memory after byte[] and char[] from
the profiler. Now Blocks has gone down in the list.
I will submit the patch today. We could wait till current trunk is more stable
to check it in.
> Reducing memory consumption on Namenode : Part 1
> ------------------------------------------------
>
> Key: HADOOP-803
> URL: https://issues.apache.org/jira/browse/HADOOP-803
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: Raghu Angadi
> Assigned To: Raghu Angadi
> Fix For: 0.11.0
>
> Attachments: block-refs-2.patch, block-refs-3.patch,
> block-refs-5.patch, HADOOP-803.patch
>
>
> There appears to be some places in Namenode that allow reducing memory
> consumption without intrusive code or feature changes. This bug is an initial
> attempt making those changes. Please include your thoughts as well.
> One change I am planning to make :
> Currently one copy of each block exists for each of the replicas and one copy
> for blockMap. I think they are all supposed to be same.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
