[
https://issues.apache.org/jira/browse/HADOOP-19140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17879167#comment-17879167
]
ASF GitHub Bot commented on HADOOP-19140:
-----------------------------------------
steveloughran commented on PR #6703:
URL: https://github.com/apache/hadoop/pull/6703#issuecomment-2328444967
@anujmodi2021
For the work on manifest committer I was asking for some IOPs per rename, so
that if there wasn't enough capacity, only those over capacity renames blocked.
It also allows for incremental IO: you don't have to block acquire up front,
just ask as you go along.
gets a bit more complex for S3 where dir operations are mimicked by
file-by-file. There nwe'd ask for 2 read and 1 write ops per file rename (HEAD
(read) + COPY (read + write) and for the bulk delete to be the same #of writes
as the delete list. That is already done in its implementation of BulkDelete.
Note that the AWS SDK does split up large COPY operations into multipart
copies, so really the IO capacity is (2 * file-size/block size) but as these
copies can be so slow I'm not worrying about it. We'd need to replace that bit
of the SDK and while we've discussed it.
FYI I've let this work lapse as other things took priority; if you want to
take it up -feel free to do so.
> [ABFS, S3A] Add IORateLimiter api to hadoop common
> --------------------------------------------------
>
> Key: HADOOP-19140
> URL: https://issues.apache.org/jira/browse/HADOOP-19140
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs, fs/azure, fs/s3
> Affects Versions: 3.4.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Labels: pull-request-available
>
> Create a rate limiter API in hadoop common which code (initially, manifest
> committer, bulk delete).. can request iO capacity for a specific operation.
> this can be exported by filesystems so support shared rate limiting across
> all threads
> pulled from HADOOP-19093 PR
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
