[
https://issues.apache.org/jira/browse/HADOOP-15460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-15460:
------------------------------------
Summary: S3A FS to add "fs.s3a.create.performance" to the builder file
creation option set (was: S3A FS to add ""fs.s3a.create.performance" to the
builder file creation option set)
> S3A FS to add "fs.s3a.create.performance" to the builder file creation
> option set
> ----------------------------------------------------------------------------------
>
> Key: HADOOP-15460
> URL: https://issues.apache.org/jira/browse/HADOOP-15460
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.1.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Major
>
> As promised to [~StephanEwen]: add and s3a-specific option to the builder-API
> to create files for all existence checks to be skipped.
> This
> # eliminates a few hundred milliseconds
> # -avoids any caching of negative HEAD/GET responses in the S3 load
> balancers.-
> Callers will be expected to know what what they are doing.
> FWIW, we are doing some PUT calls in the committer which bypass this stuff,
> for the same reason. If you've just created a directory, you know there's
> nothing underneath, so no need to check.
> adding this inside HADOOP-17833 as we are effectively doing this under the
> magic dir tree. having it as an option and using it to save all
> manifests/success files also saves one LIST per manifest write (task commit)
> and the LIST when saving a _SUCCESS file.
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
