[
https://issues.apache.org/jira/browse/YARN-1771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13923226#comment-13923226
]
Sangjin Lee commented on YARN-1771:
-----------------------------------
I have created a status cache at the LocalizerContext level, and let FSDownload
utilize the cache when querying the file status for the parent directories.
I considered using a simple synchronized map and ConcurrentHashMap, but settled
on using guava's LoadingCache. The issue with the localization pattern is that
it is bursty. Most of the downloads happen in parallel, and thus most of these
getFileStatus calls also go out in a burst. With a synchronized map, the
problem is that these calls would be unnecessarily serialized (as it needs to
acquire a global lock for this map). With a ConcurrentHashMap, calls can be
concurrent, but with a simple ConcurrentMap usage it becomes harder to avoid
extra getFileStatus calls.
The LoadingCache maintains concurrency *and* limits the getFileStatus calls to
strictly one call per path (I added the unit test to verify that).
> many getFileStatus calls made from node manager for localizing a public
> distributed cache resource
> --------------------------------------------------------------------------------------------------
>
> Key: YARN-1771
> URL: https://issues.apache.org/jira/browse/YARN-1771
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 2.3.0
> Reporter: Sangjin Lee
> Assignee: Sangjin Lee
> Priority: Critical
> Attachments: yarn-1771.patch
>
>
> We're observing that the getFileStatus calls are putting a fair amount of
> load on the name node as part of checking the public-ness for localizing a
> resource that belong in the public cache.
> We see 7 getFileStatus calls made for each of these resource. We should look
> into reducing the number of calls to the name node. One example:
> {noformat}
> 2014-02-27 18:07:27,351 INFO audit: ... cmd=getfileinfo
> src=/tmp/temp-887708724/tmp883330348/foo-0.0.44.jar ...
> 2014-02-27 18:07:27,352 INFO audit: ... cmd=getfileinfo
> src=/tmp/temp-887708724/tmp883330348/foo-0.0.44.jar ...
> 2014-02-27 18:07:27,352 INFO audit: ... cmd=getfileinfo
> src=/tmp/temp-887708724/tmp883330348 ...
> 2014-02-27 18:07:27,353 INFO audit: ... cmd=getfileinfo
> src=/tmp/temp-887708724 ...
> 2014-02-27 18:07:27,353 INFO audit: ... cmd=getfileinfo src=/tmp ...
> 2014-02-27 18:07:27,354 INFO audit: ... cmd=getfileinfo src=/ ...
> 2014-02-27 18:07:27,354 INFO audit: ... cmd=getfileinfo
> src=/tmp/temp-887708724/tmp883330348/foo-0.0.44.jar ...
> 2014-02-27 18:07:27,355 INFO audit: ... cmd=open
> src=/tmp/temp-887708724/tmp883330348/foo-0.0.44.jar ...
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
