[
https://issues.apache.org/jira/browse/IMPALA-6544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16727092#comment-16727092
]
Philip Zeyliger commented on IMPALA-6544:
-----------------------------------------
Per
https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#ConsistencyModel,
we should be experiencing read-after-write consistency:
{quote}
Amazon S3 provides read-after-write consistency for PUTS of new objects in your
S3 bucket in all regions with one caveat. The caveat is that if you make a HEAD
or GET request to the key name (to find if the object exists) before creating
the object, Amazon S3 provides eventual consistency for read-after-write.
{quote}
Perhaps we're re-using filenames across runs. Then we'd have effectively gotten
one of these HEAD/GET requests.
Or perhaps HDFS is doing a GET before writing a file.
I've not yet traced through these paths to figure out if we're hitting that.
> Lack of S3 consistency leads to rare test failures
> --------------------------------------------------
>
> Key: IMPALA-6544
> URL: https://issues.apache.org/jira/browse/IMPALA-6544
> Project: IMPALA
> Issue Type: Task
> Components: Frontend
> Affects Versions: Impala 2.8.0
> Reporter: Sailesh Mukil
> Priority: Major
> Labels: S3, broken-build, consistency, flaky, test-framework
>
> Every now and then, we hit a flaky test on S3 runs due to files missing when
> they should be present, and vice versa. We could consider running our tests
> (or a subset of our tests) with S3Guard to avoid these problems, however rare
> they are.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
