[
https://issues.apache.org/jira/browse/HADOOP-18640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17695514#comment-17695514
]
ASF GitHub Bot commented on HADOOP-18640:
-----------------------------------------
sreeb-msft opened a new pull request, #5446:
URL: https://github.com/apache/hadoop/pull/5446
This PR introduces two changes that allows client-side throttling and
backoff only for new requests, and increases the level of control through new
configs in AbfsClientThrottlingAnalyzer.
For the first change, it checks for whether the current rest operation
corresponds to a new or retried request. In case of a new request, it calls the
necessary methods to apply throttling and backoff at the client side if
necessary. In case of a retried request, these methods are skipped and no
backoff is applied, or even checked if necessary. This, however, does not
affect any other flow such as updating metrics, which happens for each request,
irrespective of whether retried or new.
In code, the check for whether it is a retried request or not, and the
subsequent call for CST is moved to a separate new method, which directly takes
input of the current retry count, and makes the decision based on that.
#### Tests Added:
Two separate tests have been added as part of this change, one for the read
and the other for write/append requests - which are the only cases where
client-side throttling is applied. Each test does the following -
1. Validates for a new request:
- The method for CST (sendingRequest) is called.
- The counters for throttling (read/append) are incremented by 1.
2. Validates for a retried request:
- The CST method (sendingRequest) call is skipped.
- The counters for throttling (read/append) are not incremented and are the
same before and after the apply throttling backoff call happens.
For the second change, new configs are introduced for the following -
1. `fs.azure.min.acceptable.error.percentage`
2. `fs.azure.max.equilibrium.error.percentage`
3. `fs.azure.rapid.sleep.decrease.factor`
4. `fs.azure.rapid.sleep.decrease.transition.ms`
5. `fs.azure.sleep.decrease.factor`
6. `fs.azure.sleep.increase.factor`
All of these are meant to replace the static values that were in place, by
allowing the user more control to configure these directly. The static values
for these variables, earlier present in code itself, now would be used as
default values for the configs.
> ABFS: Enabling Client-side Backoff only for new requests
> --------------------------------------------------------
>
> Key: HADOOP-18640
> URL: https://issues.apache.org/jira/browse/HADOOP-18640
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
> Reporter: Sree Bhattacharyya
> Assignee: Sree Bhattacharyya
> Priority: Minor
>
> Enabling backoff only for new requests that happen, and disabling for retried
> requests.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
