zhihai xu created YARN-3491:
-------------------------------
Summary: Improve the public resource localization to do both
FSDownload submission to the thread pool and completed localization handling in
one thread (PublicLocalizer).
Key: YARN-3491
URL: https://issues.apache.org/jira/browse/YARN-3491
Project: Hadoop YARN
Issue Type: Improvement
Components: nodemanager
Affects Versions: 2.7.0
Reporter: zhihai xu
Assignee: zhihai xu
Priority: Critical
Improve the public resource localization to do both FSDownload submission to
the thread pool and completed localization handling in one thread
(PublicLocalizer).
Currently FSDownload submission to the thread pool is done in
PublicLocalizer#addResource which is running in Dispatcher thread and completed
localization handling is done in PublicLocalizer#run which is running in
PublicLocalizer thread.
Because FSDownload submission to the thread pool at the following code is time
consuming, the thread pool can't be fully utilized. Instead of doing public
resource localization in parallel(multithreading), public resource localization
is serialized most of the time.
{code}
synchronized (pending) {
pending.put(queue.submit(new FSDownload(lfs, null, conf,
publicDirDestPath, resource,
request.getContext().getStatCache())),
request);
}
{code}
Also there are two more benefits with this change:
1. The Dispatcher thread won't be blocked by above FSDownload submission.
Dispatcher thread handles most of time critical events at Node manager.
2. don't need synchronization on HashMap (pending).
Because pending will be only accessed in PublicLocalizer thread.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
