[
https://issues.apache.org/jira/browse/FLINK-8402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16418879#comment-16418879
]
ASF GitHub Bot commented on FLINK-8402:
---------------------------------------
Github user NicoK commented on the issue:
https://github.com/apache/flink/pull/5624
Indeed, Presto-S3 does better in
`com.facebook.presto.hive.PrestoS3FileSystem#create()`:
```
if ((!overwrite) && exists(path)) {
throw new IOException("File already exists:" + path);
}
// file creation
```
But if `overwrite = false`, it will also check for existence first. Also,
contrary to my initial analysis, the retries when retrieving the file status
during the existence check do not cover non-existence. I can adapt the tests to
only use `overwrite = true`, but actual code outside the tests makes use of
both variants.
It's therefore a good idea to make the distinction between
`flink-s3-fs-hadoop` and `flink-s3-fs-presto` but only for the existence check,
not for checking that a file/directory was deleted since
> Amazon S3 offers eventual consistency for overwrite PUTS and DELETES in
all regions.
I adapted the code accordingly which effectively boiled down to removing
some of the new eventual consistent existence checks in
`PrestoS3FileSystemITCase`.
Regarding the two implementations you provided: for doing the existence
check, there should not be a difference between a single `fs.exists()` call vs.
`fs.open()` in terms of consistency.
> HadoopS3FileSystemITCase.testDirectoryListing fails on Travis
> -------------------------------------------------------------
>
> Key: FLINK-8402
> URL: https://issues.apache.org/jira/browse/FLINK-8402
> Project: Flink
> Issue Type: Bug
> Components: FileSystem, Tests
> Affects Versions: 1.5.0
> Reporter: Till Rohrmann
> Assignee: Nico Kruber
> Priority: Blocker
> Labels: test-stability
> Fix For: 1.5.0, 1.4.3
>
>
> The test {{HadoopS3FileSystemITCase.testDirectoryListing}} fails on Travis.
> https://travis-ci.org/tillrohrmann/flink/jobs/327021175
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
