[
https://issues.apache.org/jira/browse/HADOOP-18420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17584356#comment-17584356
]
Daniel Carl Jones commented on HADOOP-18420:
--------------------------------------------
I've had a quick look into the code... It don't see a way to mutate the request
between S3A retries by the invoker... I wonder if there's some way we can still
utilise the RetryPolicy if we had to write the retry logic using invoker's
'once' methods.
(thanks Bryan for the pointer on this optimisation)
> Optimise S3A’s recursive delete to drop successful S3 keys on retry of S3
> DeleteObjects
> ---------------------------------------------------------------------------------------
>
> Key: HADOOP-18420
> URL: https://issues.apache.org/jira/browse/HADOOP-18420
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Reporter: Daniel Carl Jones
> Priority: Major
>
> S3A users with large filesystems performing renames or deletes can run into
> throttling when S3A performs a bulk delete on keys. These are currently
> batches of 250
> ([https://github.com/apache/hadoop/blob/c1d82cd95e375410cb0dffc2931063d48687386f/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java#L319-L323]).
> When the bulk delete ([S3
> DeleteObjects|https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjects.html])
> fails, it provides a list of keys that failed and why. Today, S3A recovers
> from throttles by sending the DeleteObjects request again with no change.
> This can result in additional deletes and counts towards throttling limits.
> Instead, S3A should retry only the keys that failed, limiting the number of
> mutations against the S3 bucket, and hopefully mitigate errors when deleting
> a large number of objects.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
