[
https://issues.apache.org/jira/browse/HDFS-1366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12905412#action_12905412
]
dhruba borthakur commented on HDFS-1366:
----------------------------------------
{code}
"main" prio=10 tid=0x0000000040114800 nid=0x3f15 runnable [0x000000004022a000]
java.lang.Thread.State: RUNNABLE
at
org.apache.hadoop.hdfs.server.namenode.BlocksMap$BlockInfo.getNext(BlocksMap.java:277)
at
org.apache.hadoop.hdfs.util.LightWeightGSet.get(LightWeightGSet.java:121)
at
org.apache.hadoop.hdfs.server.namenode.BlocksMap.checkBlockInfo(BlocksMap.java:353)
at
org.apache.hadoop.hdfs.server.namenode.BlocksMap.addINode(BlocksMap.java:370)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.addToParent(FSDirectory.java:319)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:956)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:813)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:370)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:141)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:328)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:305)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:254)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:332)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1120)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1129)
{code}
The idea here is that when the fsimage is being loaded,
BlocksMap.checkBlockInfo() will never find the block in the blocksMap, i.e. the
call to blocks.get() will always return null.
> reduce namenode startup time by optimising checkBlockInfo while loading
> fsimage
> ---------------------------------------------------------------------------------
>
> Key: HDFS-1366
> URL: https://issues.apache.org/jira/browse/HDFS-1366
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: dhruba borthakur
>
> The namenode spends about 10 minutes reading in a 14 GB fsimage file into
> memory and creating all the in-memory data structures. A jstack based
> debugger clearly shows that most of the time during the fsimage load is spent
> in BlocksMap.checkBlockInfo. There is a easy way to optimize this method
> especially for this code path.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
