[
https://issues.apache.org/jira/browse/FLINK-8801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nico Kruber reopened FLINK-8801:
--------------------------------
> S3's eventual consistent read-after-write may fail yarn deployment of
> resources to S3
> -------------------------------------------------------------------------------------
>
> Key: FLINK-8801
> URL: https://issues.apache.org/jira/browse/FLINK-8801
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN, FileSystems, Runtime / Coordination
> Affects Versions: 1.4.0, 1.5.0, 1.6.4, 1.7.2, 1.8.1, 1.9.0
> Reporter: Nico Kruber
> Assignee: Nico Kruber
> Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.4.3, 1.5.0, 1.10.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> According to
> https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#ConsistencyModel:
> {quote}
> Amazon S3 provides read-after-write consistency for PUTS of new objects in
> your S3 bucket in all regions with one caveat. The caveat is that if you make
> a HEAD or GET request to the key name (to find if the object exists) before
> creating the object, Amazon S3 provides eventual consistency for
> read-after-write.
> {quote}
> Some S3 file system implementations may actually execute such a request for
> the about-to-write object and thus the read-after-write is only eventually
> consistent. {{org.apache.flink.yarn.Utils#setupLocalResource()}} currently
> relies on a consistent read-after-write since it accesses the remote resource
> to get file size and modification timestamp. Since there we have access to
> the local resource, we can use the data from there instead and circumvent the
> problem.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
