[
https://issues.apache.org/jira/browse/HADOOP-19072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17842856#comment-17842856
]
ASF GitHub Bot commented on HADOOP-19072:
-----------------------------------------
virajjasani commented on PR #6543:
URL: https://github.com/apache/hadoop/pull/6543#issuecomment-2089769498
The above proposal of providing list of optimization flags sounds impressive.
Please let me know if this summary looks good:
As part of this Jira:
- Add `fs.s3a.performance.options` as new config with only valid values as
`create` and `mkdir` for now.
- Create `S3APerformanceFlags` class (which can contain List of Enum
values). Enum can be PerformanceFlag and it should be defined in `StoreContext`.
- Mapping of the comma separated String value of
`fs.s3a.performance.options` to `S3APerformanceFlags` object can be done as
static utility of `S3APerformanceFlags` class.
- Unknown flags are logged once at info
- Provide PathCapability for `fs.s3a.performance.options.${flag}` where
${flag} value would be create/mkdir for now. When this is probed,
pathCapability should call `S3APerformanceFlags#hasCapability(${flag})`.
- Document the policy for `fs.s3a.performance.options` to indicate that the
semantic of a particular optimization flag must not change but new optimization
option could be provided in future to tune this behavior.
For future Jiras:
- Add more optimization options for `delete`, `rename` operations.
One question: IIUC, we don't need to keep the current PR behavior in case
`fs.s3a.create.performance` is enabled, since we are not introducing new
`fs.s3a.performance.options`, correct? Also, will it be prudent to deprecate
config `fs.s3a.create.performance` and perhaps log at once in s3afs if user is
still using it? Probably we can do it in separate jira too.
> S3A: expand optimisations on stores with "fs.s3a.create.performance"
> --------------------------------------------------------------------
>
> Key: HADOOP-19072
> URL: https://issues.apache.org/jira/browse/HADOOP-19072
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.4.0
> Reporter: Steve Loughran
> Assignee: Viraj Jasani
> Priority: Major
> Labels: pull-request-available
>
> on an s3a store with fs.s3a.create.performance set, speed up other operations
> * mkdir to skip parent directory check: just do a HEAD to see if there's a
> file at the target location
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
