[
https://issues.apache.org/jira/browse/HADOOP-3272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12590582#action_12590582
]
lohit vijayarenu commented on HADOOP-3272:
------------------------------------------
Here is the jmap output of namenode without the patch
{noformat}
[lohit@ ~]$jmap -histo 34713 | grep Block
8: 23347 747104 org.apache.hadoop.dfs.BlocksMap$BlockInfo
13: 23352 560448 org.apache.hadoop.dfs.Block
21: 1785 119120 [Lorg.apache.hadoop.dfs.BlocksMap$BlockInfo;
396: 5 80 [Lorg.apache.hadoop.dfs.Block;
399: 5 80 org.apache.hadoop.dfs.BlocksMap$NodeIterator
443: 4 64
org.apache.hadoop.dfs.BlockCrcUpgradeObjectNamenode$UpgradeStatus
479: 1 48 org.apache.hadoop.dfs.PendingReplicationBlocks
489: 2 48 org.apache.hadoop.dfs.LocatedBlock
516: 1 40 java.util.concurrent.LinkedBlockingQueue
586: 1 32
[Lorg.apache.hadoop.dfs.BlockCrcUpgradeObjectNamenode$UpgradeStatus;
696: 1 16 org.apache.hadoop.dfs.BlocksMap
700: 1 16 java.util.concurrent.LinkedBlockingQueue$Node
702: 1 16 org.apache.hadoop.dfs.UnderReplicatedBlocks
721: 1 16
org.apache.hadoop.dfs.PendingReplicationBlocks$PendingReplicationMonitor
838: 1 8 org.apache.hadoop.dfs.LocatedBlock$1
853: 1 8 org.apache.hadoop.dfs.LocatedBlocks$2
862: 1 8 org.apache.hadoop.dfs.BlockCommand$1
880: 1 8 org.apache.hadoop.dfs.Block$1
{noformat}
and here is one with the patch
{noformat}
[lohit@ ~]$jmap -hist 29874 | grep Block
13: 9362 299584 org.apache.hadoop.dfs.BlocksMap$BlockInfo
35: 1184 53096 [Lorg.apache.hadoop.dfs.BlocksMap$BlockInfo;
236: 27 648 org.apache.hadoop.dfs.LocatedBlock
274: 27 432 org.apache.hadoop.dfs.BlocksMap$NodeIterator
475: 3 96 org.apache.hadoop.dfs.LocatedBlocks
555: 4 64
org.apache.hadoop.dfs.BlockCrcUpgradeObjectNamenode$UpgradeStatus
600: 1 48 org.apache.hadoop.dfs.PendingReplicationBlocks
638: 1 40 java.util.concurrent.LinkedBlockingQueue
732: 1 32
[Lorg.apache.hadoop.dfs.BlockCrcUpgradeObjectNamenode$UpgradeStatus;
750: 1 24
org.apache.hadoop.dfs.PendingReplicationBlocks$PendingBlockInfo
859: 1 16 org.apache.hadoop.dfs.BlocksMap
866: 1 16 java.util.concurrent.LinkedBlockingQueue$Node
871: 1 16 org.apache.hadoop.dfs.UnderReplicatedBlocks
893: 1 16
org.apache.hadoop.dfs.PendingReplicationBlocks$PendingReplicationMonitor
953: 1 8 org.apache.hadoop.dfs.BlockCommand$1
1014: 1 8 org.apache.hadoop.dfs.LocatedBlock$1
1025: 1 8 org.apache.hadoop.dfs.LocatedBlocks$2
1046: 1 8 org.apache.hadoop.dfs.Block$1
{noformat}
Number of blocks in both run were different, but we can see that number of
difference in Block objects.
We have key/value (Block/BlockInfo) for each Block in the FileSystem stored in
FSNameSystem.blocksMap. Using the same BlockInfo for both key/value
(BlockInfo/BlockInfo) would reduce one copy of Block for each Block in
FileSystem
I will run TestDFSIO as suggested by Konstantine
> Reduce redundant copy of Block object in BlocksMap.map hash map
> ---------------------------------------------------------------
>
> Key: HADOOP-3272
> URL: https://issues.apache.org/jira/browse/HADOOP-3272
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Environment: All
> Reporter: lohit vijayarenu
> Assignee: lohit vijayarenu
> Fix For: 0.18.0
>
> Attachments: HADOOP-3272.patch
>
>
> Looks like we might have redundant copy of Block object as Key for
> BlocksMap.map hash map. We should restore this to using same object for both
> Key, Value to save space.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
