[
https://issues.apache.org/jira/browse/FLINK-8801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16811267#comment-16811267
]
Yan Yan commented on FLINK-8801:
--------------------------------
Hi [~NicoK],
I have the same concern. I was using the org.apache.hadoop.fs.s3a.S3AFileSystem
provided by the hadoop-aws package, and still got the failure with your fix.
Could you share which implementation you were using?
I was able to workaround the issue the other way round: by reading the
timestamp from S3 and override the local resource. I think it might be a better
fix since it does not reply on any S3AFileSystem methods. Could you share your
thoughts?
Thanks!
> S3's eventual consistent read-after-write may fail yarn deployment of
> resources to S3
> -------------------------------------------------------------------------------------
>
> Key: FLINK-8801
> URL: https://issues.apache.org/jira/browse/FLINK-8801
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN, FileSystems, Runtime / Coordination
> Affects Versions: 1.4.0, 1.5.0
> Reporter: Nico Kruber
> Assignee: Nico Kruber
> Priority: Blocker
> Fix For: 1.4.3, 1.5.0
>
>
> According to
> https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#ConsistencyModel:
> {quote}
> Amazon S3 provides read-after-write consistency for PUTS of new objects in
> your S3 bucket in all regions with one caveat. The caveat is that if you make
> a HEAD or GET request to the key name (to find if the object exists) before
> creating the object, Amazon S3 provides eventual consistency for
> read-after-write.
> {quote}
> Some S3 file system implementations may actually execute such a request for
> the about-to-write object and thus the read-after-write is only eventually
> consistent. {{org.apache.flink.yarn.Utils#setupLocalResource()}} currently
> relies on a consistent read-after-write since it accesses the remote resource
> to get file size and modification timestamp. Since there we have access to
> the local resource, we can use the data from there instead and circumvent the
> problem.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
