[
https://issues.apache.org/jira/browse/YARN-5289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15346075#comment-15346075
]
Steve Loughran commented on YARN-5289:
--------------------------------------
Note that this addresses some situations with filesystems and the localizer,
where it's reporting FS timing differences even when the files are unchanged.
Checking purely on timestamp , while simple, is a pretty brittle form of
validation.
BTW, looking at the code of getFileChecksum in HDFS, it can throw
{{UnsupportedFileSystemException}} if a symlink points to an FS without the
feature, whereas the API says "return null". YARN code had better check for
both when building requests; when localizing, skip checks if checksum == 0
> Use checksum in localization requests
> -------------------------------------
>
> Key: YARN-5289
> URL: https://issues.apache.org/jira/browse/YARN-5289
> Project: Hadoop YARN
> Issue Type: Bug
> Reporter: Yesha Vora
>
> Currently, timestamps is being used in localization requests. Because using
> timestamps is always pretty brittle. ( we notice a clock jump in the VM.)
> For the filesystem supporting getFileChecksum(Path), the checksum can be used
> instead of timestamps in localisation requests
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
