[jira] [Updated] (FLINK-8801) S3's eventual consistent read-after-write may fail yarn deployment of resources to S3
[
https://issues.apache.org/jira/browse/FLINK-8801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nico Kruber updated FLINK-8801:
---
Fix Version/s: (was: 1.10.0)
> S3's eventual consistent read-after-write may fail yarn deployment of
> resources to S3
> -
>
> Key: FLINK-8801
> URL: https://issues.apache.org/jira/browse/FLINK-8801
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN, FileSystems, Runtime / Coordination
>Affects Versions: 1.4.0, 1.5.0, 1.6.4, 1.7.2, 1.8.1, 1.9.0
>Reporter: Nico Kruber
>Assignee: Nico Kruber
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.4.3, 1.5.0, 1.9.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> According to
> https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#ConsistencyModel:
> {quote}
> Amazon S3 provides read-after-write consistency for PUTS of new objects in
> your S3 bucket in all regions with one caveat. The caveat is that if you make
> a HEAD or GET request to the key name (to find if the object exists) before
> creating the object, Amazon S3 provides eventual consistency for
> read-after-write.
> {quote}
> Some S3 file system implementations may actually execute such a request for
> the about-to-write object and thus the read-after-write is only eventually
> consistent. {{org.apache.flink.yarn.Utils#setupLocalResource()}} currently
> relies on a consistent read-after-write since it accesses the remote resource
> to get file size and modification timestamp. Since there we have access to
> the local resource, we can use the data from there instead and circumvent the
> problem.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (FLINK-8801) S3's eventual consistent read-after-write may fail yarn deployment of resources to S3
[
https://issues.apache.org/jira/browse/FLINK-8801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nico Kruber updated FLINK-8801:
---
Affects Version/s: 1.9.0
1.6.4
1.7.2
1.8.1
> S3's eventual consistent read-after-write may fail yarn deployment of
> resources to S3
> -
>
> Key: FLINK-8801
> URL: https://issues.apache.org/jira/browse/FLINK-8801
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN, FileSystems, Runtime / Coordination
>Affects Versions: 1.4.0, 1.5.0, 1.6.4, 1.7.2, 1.8.1, 1.9.0
>Reporter: Nico Kruber
>Assignee: Nico Kruber
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.4.3, 1.5.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> According to
> https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#ConsistencyModel:
> {quote}
> Amazon S3 provides read-after-write consistency for PUTS of new objects in
> your S3 bucket in all regions with one caveat. The caveat is that if you make
> a HEAD or GET request to the key name (to find if the object exists) before
> creating the object, Amazon S3 provides eventual consistency for
> read-after-write.
> {quote}
> Some S3 file system implementations may actually execute such a request for
> the about-to-write object and thus the read-after-write is only eventually
> consistent. {{org.apache.flink.yarn.Utils#setupLocalResource()}} currently
> relies on a consistent read-after-write since it accesses the remote resource
> to get file size and modification timestamp. Since there we have access to
> the local resource, we can use the data from there instead and circumvent the
> problem.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (FLINK-8801) S3's eventual consistent read-after-write may fail yarn deployment of resources to S3
[
https://issues.apache.org/jira/browse/FLINK-8801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-8801:
--
Labels: pull-request-available (was: )
> S3's eventual consistent read-after-write may fail yarn deployment of
> resources to S3
> -
>
> Key: FLINK-8801
> URL: https://issues.apache.org/jira/browse/FLINK-8801
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN, FileSystems, Runtime / Coordination
>Affects Versions: 1.4.0, 1.5.0
>Reporter: Nico Kruber
>Assignee: Nico Kruber
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.4.3, 1.5.0
>
>
> According to
> https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#ConsistencyModel:
> {quote}
> Amazon S3 provides read-after-write consistency for PUTS of new objects in
> your S3 bucket in all regions with one caveat. The caveat is that if you make
> a HEAD or GET request to the key name (to find if the object exists) before
> creating the object, Amazon S3 provides eventual consistency for
> read-after-write.
> {quote}
> Some S3 file system implementations may actually execute such a request for
> the about-to-write object and thus the read-after-write is only eventually
> consistent. {{org.apache.flink.yarn.Utils#setupLocalResource()}} currently
> relies on a consistent read-after-write since it accesses the remote resource
> to get file size and modification timestamp. Since there we have access to
> the local resource, we can use the data from there instead and circumvent the
> problem.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-8801) S3's eventual consistent read-after-write may fail yarn deployment of resources to S3
[
https://issues.apache.org/jira/browse/FLINK-8801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nico Kruber updated FLINK-8801:
---
Fix Version/s: 1.4.3
> S3's eventual consistent read-after-write may fail yarn deployment of
> resources to S3
> -
>
> Key: FLINK-8801
> URL: https://issues.apache.org/jira/browse/FLINK-8801
> Project: Flink
> Issue Type: Bug
> Components: FileSystem, ResourceManager, YARN
>Affects Versions: 1.4.0, 1.5.0
>Reporter: Nico Kruber
>Assignee: Nico Kruber
>Priority: Blocker
> Fix For: 1.5.0, 1.4.3
>
>
> According to
> https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#ConsistencyModel:
> {quote}
> Amazon S3 provides read-after-write consistency for PUTS of new objects in
> your S3 bucket in all regions with one caveat. The caveat is that if you make
> a HEAD or GET request to the key name (to find if the object exists) before
> creating the object, Amazon S3 provides eventual consistency for
> read-after-write.
> {quote}
> Some S3 file system implementations may actually execute such a request for
> the about-to-write object and thus the read-after-write is only eventually
> consistent. {{org.apache.flink.yarn.Utils#setupLocalResource()}} currently
> relies on a consistent read-after-write since it accesses the remote resource
> to get file size and modification timestamp. Since there we have access to
> the local resource, we can use the data from there instead and circumvent the
> problem.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
