[
https://issues.apache.org/jira/browse/HADOOP-16823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-16823:
------------------------------------
Release Note:
The page size for bulk delete operations has been reduced from 1000 to 250 to
reduce the likelihood of overloading an S3 partition, especially because the
retry policy on throttling is simply to try again.
The page size can be set in "fs.s3a.bulk.delete.page.size"
There is also an option to control whether or not the AWS client retries
requests, or whether it is handled exclusively in the S3A code. This option
"fs.s3a.experimental.aws.s3.throttling" is true by default. If set to false:
everything is handled in the S3A client. While this means that metrics may be
more accurate, it may mean that throttling failures in helper threads of the
AWS SDK (especially those used in copy/rename) may not be handled properly.
This is experimental, and should be left at "true" except when seeking more
detail about throttling rates.
was:
The AWS SDK client no longer handles 503/slow down messages from S3 with its
own internall retry mechanism; these throttling messages are handled purely in
the S3A client, which updates its counters/metrics before performing its own
backoff/retry strategy,
The values of "fs.s3a.retry.throttle.interval" and
"fs.s3a.retry.throttle.limit" have been set to compensate for the fact that the
SDK will not be retrying internally: the values are 500ms and 20 respectively.
If you have explicitly set these values. Make them larger. The default values
for the AWS SDK are defined in com.amazonaws.retry.PredefinedRetryPolicies;
currently defined as 500ms based delay + exponential/jittered backoff to 20
seconds, which is about 4-5 attempts. The S3A throttle limit has been increased
from 10 to 20 to (over) compensate. The S3A retry policy's Jitter is slightly
randomised so that multiple threads encountering throttling situations Will not
all sleep for exactly the same time. The AWS jitter seems a bit more
deterministic.
You can now inspect the metrics/statistics for a filesystem and know that this
counts the number of retries.
All other connections to S3 (especially DynamoDB) are still retried within the
AWS clients with the S3A code wrapping these.
If you are curious, consult PredefinedRetryPolicies to see what the internal
default backoff/retry policies are for S3, DynamoDB (S3Guard), etc.
> Large DeleteObject requests are their own Thundering Herd
> ---------------------------------------------------------
>
> Key: HADOOP-16823
> URL: https://issues.apache.org/jira/browse/HADOOP-16823
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.2.1
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
>
> Currently AWS S3 throttling is initially handled in the AWS SDK, only
> reaching the S3 client code after it has given up.
> This means we don't always directly observe when throttling is taking place.
> Proposed:
> * disable throttling retries in the AWS client library
> * add a quantile for the S3 throttle events, as DDB has
> * isolate counters of s3 and DDB throttle events to classify issues better
> Because we are taking over the AWS retries, we will need to expand the
> initial delay en retries and the number of retries we should support before
> giving up.
> Also: should we log throttling events? It could be useful but there is a risk
> of logs overloading especially if many threads in the same process were
> triggering the problem.
> Proposed: log at debug.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
