[
https://issues.apache.org/jira/browse/HDFS-15493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17168506#comment-17168506
]
Chengwei Wang commented on HDFS-15493:
--------------------------------------
Submit v002 patch [^HDFS-15493.002.patch].
Base on one executor with 4 threads, added a unit test, refactor code to
shutdown executor and added waiting time logging.
I had tested this patch twice:
{code:java}
Test 1.
20/07/31 14:21:17 INFO namenode.FSImageFormatPBINode: Completed loading all
INodeDirectory sub-sections
20/07/31 14:21:22 INFO namenode.FSImageFormatPBINode: Completed update
blocks map and name cache, waiting time duration(ms): 5161
20/07/31 14:21:23 INFO namenode.FSImageFormatProtobuf: Loaded FSImage in
409 seconds.
Test 2.
20/07/31 16:00:03 INFO namenode.FSImageFormatPBINode: Completed loading all
INodeDirectory sub-sections
20/07/31 16:00:16 INFO namenode.FSImageFormatPBINode: Completed update
blocks map and name cache, waiting time duration(ms): 12105
20/07/31 16:00:17 INFO namenode.FSImageFormatProtobuf: Loaded FSImage in
424 seconds.
{code}
> Update block map and name cache in parallel while loading fsimage.
> ------------------------------------------------------------------
>
> Key: HDFS-15493
> URL: https://issues.apache.org/jira/browse/HDFS-15493
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
> Reporter: Chengwei Wang
> Priority: Major
> Attachments: HDFS-15493.001.patch, HDFS-15493.002.patch,
> fsimage-loading.log
>
>
> While loading INodeDirectorySection of fsimage, it will update name cache and
> block map after added inode file to inode directory. It would reduce time
> cost of fsimage loading to enable these steps run in parallel.
> In our test case, with patch HDFS-13694 and HDFS-14617, the time cost to load
> fsimage (220M files & 240M blocks) is 470s, with this patch , the time cost
> reduc to 410s.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
