[
https://issues.apache.org/jira/browse/YARN-6315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15930205#comment-15930205
]
Jason Lowe commented on YARN-6315:
----------------------------------
I tried to run this in an end-to-end test and found it doesn't work in
practice. I was under the mistaken impression that the size specified in the
LocalResourceRequest was used to verify the correct file was being localized,
but that's not the case. It only uses the _timestamp_ to verify the correct
version of the file is being downloaded. The size is ignored. In my case the
request actually contained the value -1 for the size, so it always thought the
size mismatched and would re-localize the file. That's not good.
I thought we could pivot from the (now untrustworthy) size in the request to
the size in the LocalizedResource. That's a value the NM computes directly
during localization, so that will be correct. However this is the size of the
entire directory containing the localized resource (whether that's a file,
archive, or directory), so it includes extra things like the .crc file from
LocalFileSystem, etc. In order to match the sizes we'd have to do the same
logic being done by the localizer which is a DU of the directory. That's going
to be too expensive to do for every local resource lookup by a container launch.
> Improve LocalResourcesTrackerImpl#isResourcePresent to return false for
> corrupted files
> ---------------------------------------------------------------------------------------
>
> Key: YARN-6315
> URL: https://issues.apache.org/jira/browse/YARN-6315
> Project: Hadoop YARN
> Issue Type: Bug
> Affects Versions: 2.7.3, 2.8.1
> Reporter: Kuhu Shukla
> Assignee: Kuhu Shukla
> Attachments: YARN-6315.001.patch, YARN-6315.002.patch,
> YARN-6315.003.patch, YARN-6315.004.patch
>
>
> We currently check if a resource is present by making sure that the file
> exists locally. There can be a case where the LocalizationTracker thinks that
> it has the resource if the file exists but with size 0 or less than the
> "expected" size of the LocalResource. This JIRA tracks the change to harden
> the isResourcePresent call to address that case.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
