virajjasani commented on PR #6543:
URL: https://github.com/apache/hadoop/pull/6543#issuecomment-2089769498
The above proposal of providing list of optimization flags sounds impressive.
Please let me know if this summary looks good:
As part of this Jira:
- Add `fs.s3a.performance.options` as new config with only valid values as
`create` and `mkdir` for now.
- Create `S3APerformanceFlags` class (which can contain List of Enum
values). Enum can be PerformanceFlag and it should be defined in `StoreContext`.
- Mapping of the comma separated String value of
`fs.s3a.performance.options` to `S3APerformanceFlags` object can be done as
static utility of `S3APerformanceFlags` class.
- Unknown flags are logged once at info
- Provide PathCapability for `fs.s3a.performance.options.${flag}` where
${flag} value would be create/mkdir for now. When this is probed,
pathCapability should call `S3APerformanceFlags#hasCapability(${flag})`.
- Document the policy for `fs.s3a.performance.options` to indicate that the
semantic of a particular optimization flag must not change but new optimization
option could be provided in future to tune this behavior.
For future Jiras:
- Add more optimization options for `delete`, `rename` operations.
One question: IIUC, we don't need to keep the current PR behavior in case
`fs.s3a.create.performance` is enabled, since we are not introducing new
`fs.s3a.performance.options`, correct? Also, will it be prudent to deprecate
config `fs.s3a.create.performance` and perhaps log at once in s3afs if user is
still using it? Probably we can do it in separate jira too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
