[
https://issues.apache.org/jira/browse/HDFS-15493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17168670#comment-17168670
]
Stephen O'Donnell commented on HDFS-15493:
------------------------------------------
{quote}
After reviewed code about update blocks map and name cache carefully，I found
that it's feasible to start to do these when started loading INodeSection, and
shutdown the executors when completed loading INodeDirectorySection
{quote}
That is a good idea - I had not thought of doing this. Both the cache and block
map is working with inodes, so its strange the existing code performed these
steps in the Directory section. I will try to test performance on trunk today.
One suggestion / question - can you think of any reason someone would want to
disable this new feature? It makes the code slightly more complex to make it
optional, and I cannot really think of a reason why it would make sense to
disable it (assuming it has no bugs). I would be to remove the configuration
switch and just make it always on.
> Update block map and name cache in parallel while loading fsimage.
> ------------------------------------------------------------------
>
> Key: HDFS-15493
> URL: https://issues.apache.org/jira/browse/HDFS-15493
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
> Reporter: Chengwei Wang
> Priority: Major
> Attachments: HDFS-15493.001.patch, HDFS-15493.002.patch,
> HDFS-15493.003.patch, HDFS-15493.004.patch, fsimage-loading.log
>
>
> While loading INodeDirectorySection of fsimage, it will update name cache and
> block map after added inode file to inode directory. It would reduce time
> cost of fsimage loading to enable these steps run in parallel.
> In our test case, with patch HDFS-13694 and HDFS-14617, the time cost to load
> fsimage (220M files & 240M blocks) is 470s, with this patch , the time cost
> reduc to 410s.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
