GitHub user NicoK opened a pull request:
https://github.com/apache/flink/pull/5601
[FLINK-8336][yarn/s3][tests] harden YarnFileStageTest upload test for
eventual consistent read-after-write
## What is the purpose of the change
In case the newly written object cannot be read (yet, e.g. due to
eventually consistent read-after-write file systems like S3), we do 4 more
retries to retrieve the value and wait 50ms each. While this does not solve all
the cases it should make the (rare) case of the written object not being
available for read even more unlikely.
## Brief change log
- let `YarnFileStageTest` try 5 times waiting 50ms each before giving up on
retrieving a file that should have been uploaded before
## Verifying this change
This change is a trivial rework / code cleanup without any test coverage.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): **no**
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: **no**
- The serializers: **no**
- The runtime per-record code paths (performance sensitive): **no**
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Yarn/Mesos, ZooKeeper: **no**
- The S3 file system connector: **no**
## Documentation
- Does this pull request introduce a new feature? **no**
- If yes, how is the feature documented? **not applicable**
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/NicoK/flink flink-8336
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/5601.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5601
----
commit 9d7293fa7419344955e35851895a992af9eb09e4
Author: Nico Kruber <nico@...>
Date: 2018-02-27T16:29:00Z
[FLINK-8336][yarn/s3][tests] harden YarnFileStageTest upload test for
eventual consistent read-after-write
In case the newly written object cannot be read (yet), we do 4 more retries
to
retrieve the value and wait 50ms each. While this does not solve all the
cases
it should make the (rare) case of the written object not being available for
read even more unlikely.
----
---
