[
https://issues.apache.org/jira/browse/YARN-6315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15906665#comment-15906665
]
Kuhu Shukla edited comment on YARN-6315 at 3/12/17 7:36 PM:
------------------------------------------------------------
Some performance numbers from instrumenting the test and profiling it through
YourKit on my Macbook Pro.
The current patch spends an average of 1900 ms for 10,002 runs (189
micro-seconds per call).
An equivalent patch that uses file.isDirectory(), file.exists(), file.length()
as shown below takes 2080.8 ms for 10,002 runs (208 micro seconds per call).
{code}
if ((!file.isDirectory() && file.length() != req.getSize()) || !file.exists())
{
ret = false;
} else if (dirsHandler != null) {
ret = checkLocalResource(rsrc);
}
{code}
was (Author: kshukla):
Some performance numbers from instrumenting the test and profiling it through
YourKit on my Macbook Pro.
The current patch spends an average of 1900 ms for 10,002 runs (189
micro-seconds per call).
An equivalent patch that uses file.isDirectory(), file.exists(), file.length()
as shown below takes 2080.8 ms for 10,002 runs (0.208 micro seconds per call).
{code}
if ((!file.isDirectory() && file.length() != req.getSize()) || !file.exists())
{
ret = false;
} else if (dirsHandler != null) {
ret = checkLocalResource(rsrc);
}
{code}
> Improve LocalResourcesTrackerImpl#isResourcePresent to return false for
> corrupted files
> ---------------------------------------------------------------------------------------
>
> Key: YARN-6315
> URL: https://issues.apache.org/jira/browse/YARN-6315
> Project: Hadoop YARN
> Issue Type: Bug
> Affects Versions: 2.7.3, 2.8.1
> Reporter: Kuhu Shukla
> Assignee: Kuhu Shukla
> Attachments: YARN-6315.001.patch
>
>
> We currently check if a resource is present by making sure that the file
> exists locally. There can be a case where the LocalizationTracker thinks that
> it has the resource if the file exists but with size 0 or less than the
> "expected" size of the LocalResource. This JIRA tracks the change to harden
> the isResourcePresent call to address that case.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
