[
https://issues.apache.org/jira/browse/HDFS-15493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17167831#comment-17167831
]
Stephen O'Donnell edited comment on HDFS-15493 at 7/30/20, 11:20 AM:
---------------------------------------------------------------------
{quote}
So, awaitTermination 1 ms would make executor shutdown quickly.
{quote}
I believe if you specify a timeout of 500ms, and the threads all finish in 5ms,
the call will return after 5ms. Therefore setting it to 500 or 1000ms and
logging a message each time around the loop should not give any time penalty,
but should give us some information about what is happening.
{quote}
with the same fsimage, the time cost would increase to 430s with about 10s+
time to wait two executors shutdown.
{quote}
How long does the shutdown take with the single 4 thread executor? I cannot see
how multiple threads help, as both the methods have a lock right at the start.
If multiple threads make it faster, then it would suggest the time taken to
pick the task from the queue and start it running is significant.
Are you testing this on the trunk code + this patch, or a different version
plus this patch?
Could you try testing 2 executors with 2 threads each?
was (Author: sodonnell):
{quote}
So, awaitTermination 1 ms would make executor shutdown quickly.
{quote}
I believe if you specify a timeout of 500ms, and the threads all finish in 5ms,
the call will return. Therefore setting it to 500 or 1000ms and logging a
message each time around the loop should not give any time penalty, but should
give us some information about what is happening.
{quote}
with the same fsimage, the time cost would increase to 430s with about 10s+
time to wait two executors shutdown.
{quote}
How long does the shutdown take with the single 4 thread executor? I cannot see
how multiple threads help, as both the methods have a lock right at the start.
If multiple threads make it faster, then it would suggest the time taken to
pick the task from the queue and start it running is significant.
Are you testing this on the trunk code + this patch, or a different version
plus this patch?
Could you try testing 2 executors with 2 threads each?
> Update block map and name cache in parallel while loading fsimage.
> ------------------------------------------------------------------
>
> Key: HDFS-15493
> URL: https://issues.apache.org/jira/browse/HDFS-15493
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
> Reporter: Chengwei Wang
> Priority: Major
> Attachments: HDFS-15493.001.patch, fsimage-loading.log
>
>
> While loading INodeDirectorySection of fsimage, it will update name cache and
> block map after added inode file to inode directory. It would reduce time
> cost of fsimage loading to enable these steps run in parallel.
> In our test case, with patch HDFS-13694 and HDFS-14617, the time cost to load
> fsimage (220M files & 240M blocks) is 470s, with this patch , the time cost
> reduc to 410s.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
