[
https://issues.apache.org/jira/browse/HADOOP-15460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16476003#comment-16476003
]
Stephan Ewen commented on HADOOP-15460:
---------------------------------------
The discussion was motivated by downstream consumers of the S3AFileSystem that
like all the tooling around security, retries, multi-part uploads etc.
We would like to opt out of the consistency implications from trying to mimic a
directory structure, and rather use it more blob-store like, meaning a path is
simply a key, and it is not trying to check that no parents exist as files.
> S3A FS to add "s3a:no-existence-checks" to the builder file creation option
> set
> --------------------------------------------------------------------------------
>
> Key: HADOOP-15460
> URL: https://issues.apache.org/jira/browse/HADOOP-15460
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.1.0
> Reporter: Steve Loughran
> Priority: Major
>
> As promised to [~StephanEwen]: add and s3a-specific option to the builder-API
> to create files for all existence checks to be skipped.
> This
> # eliminates a few hundred milliseconds
> # avoids any caching of negative HEAD/GET responses in the S3 load balancers.
> Callers will be expected to know what what they are doing.
> FWIW, we are doing some PUT calls in the committer which bypass this stuff,
> for the same reason. If you've just created a directory, you know there's
> nothing underneath, so no need to check.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
