Steve Loughran created HADOOP-16644:
---------------------------------------
Summary: Intermittent failure of ITestS3ATerasortOnS3A: timestamp
differences
Key: HADOOP-16644
URL: https://issues.apache.org/jira/browse/HADOOP-16644
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3, test
Affects Versions: 3.3.0
Environment: -Dparallel-tests -DtestsThreadCount=8
-Dfailsafe.runOrder=balanced -Ds3guard -Ddynamo -Dscale
h2. Hypothesis:
the timestamp of the source file is being picked up from S3Guard, but when the
NM does a getFileStatus call, a HEAD check is made -and this (due to the
overloaded test system) is out of sync with the listing. S3Guard is updated,
the corrected date returned and the localisation fails.
Reporter: Steve Loughran
Terasort of directory committer failing in resource localisaton -the
partitions.lst file has a different TS from that expected
Happens under loaded integration tests (threads = 8; not standalone); non-auth
s3guard
{code}
2019-10-08 11:50:29,774 [IPC Server handler 4 on 55983] WARN
localizer.ResourceLocalizationService
(ResourceLocalizationService.java:processHeartbeat(1150)) - {
s3a://hwdev-steve-ireland-new/terasort-directory/sortout/_partition.lst,
1570531828143, FILE, null } failed: Resource
s3a://hwdev-steve-ireland-new/terasort-directory/sortout/_partition.lst changed
on src filesystem (expected 1570531828143, was 1570531828000
java.io.IOException: Resource
s3a://hwdev-steve-ireland-new/terasort-directory/sortout/_partition.lst changed
on src filesystem (expected 1570531828143, was 1570531828000
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
