[
https://issues.apache.org/jira/browse/YARN-6315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16055849#comment-16055849
]
Kuhu Shukla commented on YARN-6315:
-----------------------------------
Adding an 'actualSize' for LocalizedResource and then checking it against the
file attributes in isResourcePresent() covers a subset of corruption scenarios.
That is if the file size changes after its successful download. I am leaning
towards adding actualSize to reflect the "hdfs resource size" and compare that
with the local file size. This will cover any corruption caused during
download. Special case here would be directories and archives.
> Improve LocalResourcesTrackerImpl#isResourcePresent to return false for
> corrupted files
> ---------------------------------------------------------------------------------------
>
> Key: YARN-6315
> URL: https://issues.apache.org/jira/browse/YARN-6315
> Project: Hadoop YARN
> Issue Type: Bug
> Affects Versions: 2.7.3, 2.8.1
> Reporter: Kuhu Shukla
> Assignee: Kuhu Shukla
> Attachments: YARN-6315.001.patch, YARN-6315.002.patch,
> YARN-6315.003.patch, YARN-6315.004.patch
>
>
> We currently check if a resource is present by making sure that the file
> exists locally. There can be a case where the LocalizationTracker thinks that
> it has the resource if the file exists but with size 0 or less than the
> "expected" size of the LocalResource. This JIRA tracks the change to harden
> the isResourcePresent call to address that case.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
