[ 
https://issues.apache.org/jira/browse/YARN-1771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer updated YARN-1771:
-----------------------------------
    Fix Version/s:     (was: 2.5.0)
                       (was: 3.0.0)

> many getFileStatus calls made from node manager for localizing a public 
> distributed cache resource
> --------------------------------------------------------------------------------------------------
>
>                 Key: YARN-1771
>                 URL: https://issues.apache.org/jira/browse/YARN-1771
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: nodemanager
>    Affects Versions: 2.3.0
>            Reporter: Sangjin Lee
>            Assignee: Sangjin Lee
>            Priority: Critical
>             Fix For: 2.4.0
>
>         Attachments: yarn-1771.patch, yarn-1771.patch, yarn-1771.patch, 
> yarn-1771.patch
>
>
> We're observing that the getFileStatus calls are putting a fair amount of 
> load on the name node as part of checking the public-ness for localizing a 
> resource that belong in the public cache.
> We see 7 getFileStatus calls made for each of these resource. We should look 
> into reducing the number of calls to the name node. One example:
> {noformat}
> 2014-02-27 18:07:27,351 INFO audit: ... cmd=getfileinfo       
> src=/tmp/temp-887708724/tmp883330348/foo-0.0.44.jar ...
> 2014-02-27 18:07:27,352 INFO audit: ... cmd=getfileinfo       
> src=/tmp/temp-887708724/tmp883330348/foo-0.0.44.jar ...
> 2014-02-27 18:07:27,352 INFO audit: ... cmd=getfileinfo       
> src=/tmp/temp-887708724/tmp883330348 ...
> 2014-02-27 18:07:27,353 INFO audit: ... cmd=getfileinfo       
> src=/tmp/temp-887708724 ...
> 2014-02-27 18:07:27,353 INFO audit: ... cmd=getfileinfo       src=/tmp ...
> 2014-02-27 18:07:27,354 INFO audit: ... cmd=getfileinfo       src=/    ...
> 2014-02-27 18:07:27,354 INFO audit: ... cmd=getfileinfo       
> src=/tmp/temp-887708724/tmp883330348/foo-0.0.44.jar ...
> 2014-02-27 18:07:27,355 INFO audit: ... cmd=open      
> src=/tmp/temp-887708724/tmp883330348/foo-0.0.44.jar ...
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to