[
https://issues.apache.org/jira/browse/HADOOP-15460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-15460:
------------------------------------
Description:
As promised to [~StephanEwen]: add and s3a-specific option to the builder-API
to create files for all existence checks to be skipped.
This
# eliminates a few hundred milliseconds
# -avoids any caching of negative HEAD/GET responses in the S3 load balancers.-
Callers will be expected to know what what they are doing.
FWIW, we are doing some PUT calls in the committer which bypass this stuff, for
the same reason. If you've just created a directory, you know there's nothing
underneath, so no need to check.
adding this inside HADOOP-17833 as we are effectively doing this under the
magic dir tree. having it as an option and using it to save all
manifests/success files also saves one LIST per manifest write (task commit)
and the LIST when saving a _SUCCESS file.
was:
As promised to [~StephanEwen]: add and s3a-specific option to the builder-API
to create files for all existence checks to be skipped.
This
# eliminates a few hundred milliseconds
# avoids any caching of negative HEAD/GET responses in the S3 load balancers.
Callers will be expected to know what what they are doing.
FWIW, we are doing some PUT calls in the committer which bypass this stuff, for
the same reason. If you've just created a directory, you know there's nothing
underneath, so no need to check.
> S3A FS to add ""fs.s3a.create.performance" to the builder file creation
> option set
> -----------------------------------------------------------------------------------
>
> Key: HADOOP-15460
> URL: https://issues.apache.org/jira/browse/HADOOP-15460
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.1.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Major
>
> As promised to [~StephanEwen]: add and s3a-specific option to the builder-API
> to create files for all existence checks to be skipped.
> This
> # eliminates a few hundred milliseconds
> # -avoids any caching of negative HEAD/GET responses in the S3 load
> balancers.-
> Callers will be expected to know what what they are doing.
> FWIW, we are doing some PUT calls in the committer which bypass this stuff,
> for the same reason. If you've just created a directory, you know there's
> nothing underneath, so no need to check.
> adding this inside HADOOP-17833 as we are effectively doing this under the
> magic dir tree. having it as an option and using it to save all
> manifests/success files also saves one LIST per manifest write (task commit)
> and the LIST when saving a _SUCCESS file.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
