[
https://issues.apache.org/jira/browse/HDFS-15493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chengwei Wang updated HDFS-15493:
---------------------------------
Description:
While loading INodeDirectorySection of fsimage, it will update name cache and
block map after added inode file to inode directory. It would reduce time cost
of fsimage loading to enable these steps run in parallel.
In our test case, with patch HDFS-13694 and HDFS-14617, the time cost to load
fsimage (220M files & 240M blocks) is 470s, with this patch , the time cost
reduce to 410s.
was:
While loading INodeDirectorySection of fsimage, it will update name cache and
block map after added inode file to inode directory. It would reduce time cost
of fsimage loading to enable these steps run in parallel.
In our test case, with patch HDFS-13694 and HDFS-14617, the time cost to load
fsimage (220M files & 240M blocks) is 470s, with this patch , the time cost
reduc to 410s.
> Update block map and name cache in parallel while loading fsimage.
> ------------------------------------------------------------------
>
> Key: HDFS-15493
> URL: https://issues.apache.org/jira/browse/HDFS-15493
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
> Reporter: Chengwei Wang
> Assignee: Chengwei Wang
> Priority: Major
> Attachments: HDFS-15493.001.patch, HDFS-15493.002.patch,
> HDFS-15493.003.patch, HDFS-15493.004.patch, HDFS-15493.005.patch,
> HDFS-15493.006.patch, HDFS-15493.007.patch, HDFS-15493.008.patch,
> fsimage-loading.log
>
>
> While loading INodeDirectorySection of fsimage, it will update name cache and
> block map after added inode file to inode directory. It would reduce time
> cost of fsimage loading to enable these steps run in parallel.
> In our test case, with patch HDFS-13694 and HDFS-14617, the time cost to load
> fsimage (220M files & 240M blocks) is 470s, with this patch , the time cost
> reduce to 410s.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
