[
https://issues.apache.org/jira/browse/HDFS-15493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17167604#comment-17167604
]
Chengwei Wang edited comment on HDFS-15493 at 7/30/20, 3:01 AM:
----------------------------------------------------------------
Hi [~sodonnell]，sorry for missing some messages.
{quote}I mis-understood how it worked, as I thought `awaitTermination(...)`
threw an exception after the timeout, which is not the case.
{quote}
I guess you got a misunderstanding about `awaitTermination(...)` . It just like
object.wait(long time), wolud just stop blocking rather than throw a
InterruptedException.
{code:java}
/**
* Blocks until all tasks have completed execution after a shutdown
* request, or the timeout occurs, or the current thread is
* interrupted, whichever happens first.
*
* @param timeout the maximum time to wait
* @param unit the time unit of the timeout argument
* @return {@code true} if this executor terminated and
* {@code false} if the timeout elapsed before termination
* @throws InterruptedException if interrupted while waiting
*/
boolean awaitTermination(long timeout, TimeUnit unit)
throws InterruptedException;
{code}
So, awaitTermination 1 ms would make executor shutdown quickly.
{quote}Did you find the runtime was about the same with a single executor with
4 threads and two executors with a single thread?As my testing showed a small
improvement with the two single threaded executors case, and as locking
prevents more than one thread to run concurrently, I think it would be better
to go with the two executors with a single thread.
{quote}
I understand what you mean about the runtime. Intuitively, using two single
thread executors would perform better than one fiexd threads executor. But I
had tested update blocks map and cache name by two single thread executor and
removed the lock yesterday after reply, with the same fsimage, the time cost
would increase to 430s with about 10s+ time to wait two executors shutdown.
So, I'm not sure that using two single thread executors would perfrom better.
For more info, our fsimage had few snapshot, loading fsimage finished as soon
as loadINodeDirectorySection finished. In other word, delay to shutdown the
executors wouldn't work better.
was (Author: smarthan):
Hi [~sodonnell]，sorry for missing some messages.
{quote}I mis-understood how it worked, as I thought `awaitTermination(...)`
threw an exception after the timeout, which is not the case.
{quote}
I guess you got a misunderstanding about `awaitTermination(...)` . It just like
object.wait(long time), wolud just stop blocking rather than throw a
InterruptedException.
{code:java}
/**
* Blocks until all tasks have completed execution after a shutdown
* request, or the timeout occurs, or the current thread is
* interrupted, whichever happens first.
*
* @param timeout the maximum time to wait
* @param unit the time unit of the timeout argument
* @return {@code true} if this executor terminated and
* {@code false} if the timeout elapsed before termination
* @throws InterruptedException if interrupted while waiting
*/
boolean awaitTermination(long timeout, TimeUnit unit)
throws InterruptedException;
{code}
So, awaitTermination 1 ms would make executor shutdown quickly.
{quote}Did you find the runtime was about the same with a single executor with
4 threads and two executors with a single thread?As my testing showed a small
improvement with the two single threaded executors case, and as locking
prevents more than one thread to run concurrently, I think it would be better
to go with the two executors with a single thread.
{quote}
I understand what you mean about the runtime. Intuitively, using two single
thread executors would perform better than one fiexd threads executor. But I
had tested update blocks map and cache name by two single thread executor and
removed the lock yesterday after reply, with the same fsimage, the time cost
would increase to 430s with about 10s+ time to wait two executors shutdown.
So, I'm not sure that using two single thread executors would perfrom better.
For more info, our fsimage had few snapshot, loading fsimage finished as soon
as loadINodeDirectorySection finished. In other word, delay to shutdown the
executors wouldn't work better.
> Update block map and name cache in parallel while loading fsimage.
> ------------------------------------------------------------------
>
> Key: HDFS-15493
> URL: https://issues.apache.org/jira/browse/HDFS-15493
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
> Reporter: Chengwei Wang
> Priority: Major
> Attachments: HDFS-15493.001.patch, fsimage-loading.log
>
>
> While loading INodeDirectorySection of fsimage, it will update name cache and
> block map after added inode file to inode directory. It would reduce time
> cost of fsimage loading to enable these steps run in parallel.
> In our test case, with patch HDFS-13694 and HDFS-14617, the time cost to load
> fsimage (220M files & 240M blocks) is 470s, with this patch , the time cost
> reduc to 410s.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
