[ https://issues.apache.org/jira/browse/YARN-6315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15930205#comment-15930205 ]
Jason Lowe commented on YARN-6315: ---------------------------------- I tried to run this in an end-to-end test and found it doesn't work in practice. I was under the mistaken impression that the size specified in the LocalResourceRequest was used to verify the correct file was being localized, but that's not the case. It only uses the _timestamp_ to verify the correct version of the file is being downloaded. The size is ignored. In my case the request actually contained the value -1 for the size, so it always thought the size mismatched and would re-localize the file. That's not good. I thought we could pivot from the (now untrustworthy) size in the request to the size in the LocalizedResource. That's a value the NM computes directly during localization, so that will be correct. However this is the size of the entire directory containing the localized resource (whether that's a file, archive, or directory), so it includes extra things like the .crc file from LocalFileSystem, etc. In order to match the sizes we'd have to do the same logic being done by the localizer which is a DU of the directory. That's going to be too expensive to do for every local resource lookup by a container launch. > Improve LocalResourcesTrackerImpl#isResourcePresent to return false for > corrupted files > --------------------------------------------------------------------------------------- > > Key: YARN-6315 > URL: https://issues.apache.org/jira/browse/YARN-6315 > Project: Hadoop YARN > Issue Type: Bug > Affects Versions: 2.7.3, 2.8.1 > Reporter: Kuhu Shukla > Assignee: Kuhu Shukla > Attachments: YARN-6315.001.patch, YARN-6315.002.patch, > YARN-6315.003.patch, YARN-6315.004.patch > > > We currently check if a resource is present by making sure that the file > exists locally. There can be a case where the LocalizationTracker thinks that > it has the resource if the file exists but with size 0 or less than the > "expected" size of the LocalResource. This JIRA tracks the change to harden > the isResourcePresent call to address that case. -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org