[
https://issues.apache.org/jira/browse/HADOOP-15460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16476095#comment-16476095
]
Steve Loughran commented on HADOOP-15460:
-----------------------------------------
we'd skip both the checks at the beginning, and any DELETE calls put upstream
at the end. For S3Guard we still want to update the DDB tables, as long as the
cost is low. Stephan is really motivated by the problem of "writing small
checkpoint files every few seconds"; there's too much overhead around the PUT
for their code righ tnow.
> S3A FS to add "s3a:no-existence-checks" to the builder file creation option
> set
> --------------------------------------------------------------------------------
>
> Key: HADOOP-15460
> URL: https://issues.apache.org/jira/browse/HADOOP-15460
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.1.0
> Reporter: Steve Loughran
> Priority: Major
>
> As promised to [~StephanEwen]: add and s3a-specific option to the builder-API
> to create files for all existence checks to be skipped.
> This
> # eliminates a few hundred milliseconds
> # avoids any caching of negative HEAD/GET responses in the S3 load balancers.
> Callers will be expected to know what what they are doing.
> FWIW, we are doing some PUT calls in the committer which bypass this stuff,
> for the same reason. If you've just created a directory, you know there's
> nothing underneath, so no need to check.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
