[
https://issues.apache.org/jira/browse/YARN-1771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13918112#comment-13918112
]
Jason Lowe commented on YARN-1771:
----------------------------------
Here's a thought to possibly avoid checking each directory level individually:
what if the NM simply tried to read the file as the user requesting it to be
public? The NM should already have the necessary tokens to access the
resource, so it should be able to use doAs to read the file as the requesting
user. The rationale for this approach being that if the user can read the
resource and is asking for it to be public then they can trivially make the
data public themselves by copying to /tmp and make the copy publicly accessible.
> many getFileStatus calls made from node manager for localizing a public
> distributed cache resource
> --------------------------------------------------------------------------------------------------
>
> Key: YARN-1771
> URL: https://issues.apache.org/jira/browse/YARN-1771
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Reporter: Sangjin Lee
> Assignee: Sangjin Lee
> Priority: Critical
>
> We're observing that the getFileStatus calls are putting a fair amount of
> load on the name node as part of checking the public-ness for localizing a
> resource that belong in the public cache.
> We see 7 getFileStatus calls made for each of these resource. We should look
> into reducing the number of calls to the name node. One example:
> {noformat}
> 2014-02-27 18:07:27,351 INFO audit: ... cmd=getfileinfo
> src=/tmp/temp-887708724/tmp883330348/foo-0.0.44.jar ...
> 2014-02-27 18:07:27,352 INFO audit: ... cmd=getfileinfo
> src=/tmp/temp-887708724/tmp883330348/foo-0.0.44.jar ...
> 2014-02-27 18:07:27,352 INFO audit: ... cmd=getfileinfo
> src=/tmp/temp-887708724/tmp883330348 ...
> 2014-02-27 18:07:27,353 INFO audit: ... cmd=getfileinfo
> src=/tmp/temp-887708724 ...
> 2014-02-27 18:07:27,353 INFO audit: ... cmd=getfileinfo src=/tmp ...
> 2014-02-27 18:07:27,354 INFO audit: ... cmd=getfileinfo src=/ ...
> 2014-02-27 18:07:27,354 INFO audit: ... cmd=getfileinfo
> src=/tmp/temp-887708724/tmp883330348/foo-0.0.44.jar ...
> 2014-02-27 18:07:27,355 INFO audit: ... cmd=open
> src=/tmp/temp-887708724/tmp883330348/foo-0.0.44.jar ...
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
