[
https://issues.apache.org/jira/browse/HDFS-7784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14321075#comment-14321075
]
Kai Zheng commented on HDFS-7784:
---------------------------------
Hi [~walter.k.su],
It's interesting, thanks !
bq.So I changed the priority to minor
I don't think it's minor. It does make sense. I thought it's a good discussion.
bq.One thing we might consider is a two-thread system, where one thread does
deserialization and puts the results into a BlockingQueue read by the other FSN
loading thread.
I thought it's a good idea. We might consider it as well and have a try ?
So we have the current approach, the parallel approach proposed here, and the
above one suggested by [~cmccabe]. Is it possible to enhance and allow to
plugin the fsimage loading approach ? By default it will use the current method.
> load fsimage in parallel
> ------------------------
>
> Key: HDFS-7784
> URL: https://issues.apache.org/jira/browse/HDFS-7784
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
> Reporter: Walter Su
> Assignee: Walter Su
> Priority: Minor
> Attachments: HDFS-7784.001.patch, test-20150213.pdf
>
>
> When single Namenode has huge amount of files, without using federation, the
> startup/restart speed is slow. The fsimage loading step takes the most of the
> time. fsimage loading can seperate to two parts, deserialization and object
> construction(mostly map insertion). Deserialization takes the most of CPU
> time. So we can do deserialization in parallel, and add to hashmap in serial.
> It will significantly reduce the NN start time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
