Nico Kruber created FLINK-8801:
----------------------------------
Summary: S3's eventual consistent read-after-write may fail yarn
deployment of resources to S3
Key: FLINK-8801
URL: https://issues.apache.org/jira/browse/FLINK-8801
Project: Flink
Issue Type: Bug
Components: FileSystem, ResourceManager, YARN
Affects Versions: 1.4.0, 1.5.0
Reporter: Nico Kruber
Assignee: Nico Kruber
Fix For: 1.5.0
According to
https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#ConsistencyModel:
{quote}
Amazon S3 provides read-after-write consistency for PUTS of new objects in your
S3 bucket in all regions with one caveat. The caveat is that if you make a HEAD
or GET request to the key name (to find if the object exists) before creating
the object, Amazon S3 provides eventual consistency for read-after-write.
{quote}
Some S3 file system implementations may actually execute such a request for the
about-to-write object and thus the read-after-write is only eventually
consistent. {{org.apache.flink.yarn.Utils#setupLocalResource()}} currently
relies on a consistent read-after-write since it accesses the remote resource
to get file size and modification timestamp. Since there we have access to the
local resource, we can use the data from there instead and circumvent the
problem.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
