[
https://issues.apache.org/jira/browse/YARN-3491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhihai xu updated YARN-3491:
----------------------------
Description:
Based on the profiling, The bottleneck in PublicLocalizer#addResource is
getInitializedLocalDirs. getInitializedLocalDirs call checkLocalDir.
checkLocalDir is very slow which is about 10 ms.
The total delay will be approximately number of local dirs * 10 ms.
This delay will be added for each public resource localization.
It will cause public resource localization is serialized most of the time.
was:
Improve the public resource localization to do both FSDownload submission to
the thread pool and completed localization handling in one thread
(PublicLocalizer).
Currently FSDownload submission to the thread pool is done in
PublicLocalizer#addResource which is running in Dispatcher thread and completed
localization handling is done in PublicLocalizer#run which is running in
PublicLocalizer thread.
Because PublicLocalizer#addResource is time consuming, the thread pool can't be
fully utilized. Instead of doing public resource localization in
parallel(multithreading), public resource localization is serialized most of
the time.
Also there are two more benefits with this change:
1. The Dispatcher thread won't be blocked by PublicLocalizer#addResource .
Dispatcher thread handles most of time critical events at Node manager.
2. don't need synchronization on HashMap (pending).
Because pending will be only accessed in PublicLocalizer thread.
> PublicLocalizer#addResource is too slow.
> ----------------------------------------
>
> Key: YARN-3491
> URL: https://issues.apache.org/jira/browse/YARN-3491
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
> Affects Versions: 2.7.0
> Reporter: zhihai xu
> Assignee: zhihai xu
> Priority: Critical
> Attachments: YARN-3491.000.patch
>
>
> Based on the profiling, The bottleneck in PublicLocalizer#addResource is
> getInitializedLocalDirs. getInitializedLocalDirs call checkLocalDir.
> checkLocalDir is very slow which is about 10 ms.
> The total delay will be approximately number of local dirs * 10 ms.
> This delay will be added for each public resource localization.
> It will cause public resource localization is serialized most of the time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
