[
https://issues.apache.org/jira/browse/HADOOP-17306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17227777#comment-17227777
]
Steve Loughran commented on HADOOP-17306:
-----------------------------------------
bq. Note, it looks like pre-commit build only tested hadoop-common, so I can
see why the YARN failures were missed. Not sure why we don't run tests for all
projects when we make changes in common.
* patches would take forever
* things like HDFS tests fail so consistently we'd end up ignoring them
Generally, when a patch to hadoop-common goes near a module you know about,
adding a change to the PR like a newline in a comment in a test file is
sufficient to get it included -you just omit that change when you do the final
merge.
Problem here looks like the timestamp checking of yarn localiser -there's
clearly a timestamp mismatch between when the file was uploaded and the actual
time of the file.
FWIW, object stores have been brittle for this in the past, especially s3 +
S3guard...under heavy test load things can be off by a second). I've never
worried about this (intermittent) failure, as going near the localizer was
something I was worried about. Maybe: filesystems can declare their
granularity, or have their own "has equal timestamp" probe. Or YARN localizer
only worries about second granularity for its comparisions? I'd worry about the
security risks on HDFS though.
Ideally: checksums should be used for any FS which supports checksums. Of
course, S3A turns off mapping of etag -> checksum as then distcp from hdfs to
s3 fails unless you explicitly {{-skipCrcCheck}} the operation...
> RawLocalFileSystem's lastModifiedTime() looses milli seconds in JDK < 10.b09
> ----------------------------------------------------------------------------
>
> Key: HADOOP-17306
> URL: https://issues.apache.org/jira/browse/HADOOP-17306
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Reporter: Vinayakumar B
> Assignee: Vinayakumar B
> Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> RawLocalFileSystem's FileStatus uses {{File.lastModified()}} api from JDK.
> This api looses milliseconds due to JDK bug.
> [https://bugs.java.com/bugdatabase/view_bug.do?bug_id=8177809]
> This bug fixed in JDK 10 b09 onwards and still exists in JDK 8 which is still
> being used in many productions.
> Apparently, {{Files.getLastModifiedTime()}} from java's nio package returns
> correct time.
> Use {{Files.getLastModifiedTime()}} instead of {{File.lastModified}} as
> workaround.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
