sreeb-msft opened a new pull request, #5446: URL: https://github.com/apache/hadoop/pull/5446
This PR introduces two changes that allows client-side throttling and backoff only for new requests, and increases the level of control through new configs in AbfsClientThrottlingAnalyzer. For the first change, it checks for whether the current rest operation corresponds to a new or retried request. In case of a new request, it calls the necessary methods to apply throttling and backoff at the client side if necessary. In case of a retried request, these methods are skipped and no backoff is applied, or even checked if necessary. This, however, does not affect any other flow such as updating metrics, which happens for each request, irrespective of whether retried or new. In code, the check for whether it is a retried request or not, and the subsequent call for CST is moved to a separate new method, which directly takes input of the current retry count, and makes the decision based on that. #### Tests Added: Two separate tests have been added as part of this change, one for the read and the other for write/append requests - which are the only cases where client-side throttling is applied. Each test does the following - 1. Validates for a new request: - The method for CST (sendingRequest) is called. - The counters for throttling (read/append) are incremented by 1. 2. Validates for a retried request: - The CST method (sendingRequest) call is skipped. - The counters for throttling (read/append) are not incremented and are the same before and after the apply throttling backoff call happens. For the second change, new configs are introduced for the following - 1. `fs.azure.min.acceptable.error.percentage` 2. `fs.azure.max.equilibrium.error.percentage` 3. `fs.azure.rapid.sleep.decrease.factor` 4. `fs.azure.rapid.sleep.decrease.transition.ms` 5. `fs.azure.sleep.decrease.factor` 6. `fs.azure.sleep.increase.factor` All of these are meant to replace the static values that were in place, by allowing the user more control to configure these directly. The static values for these variables, earlier present in code itself, now would be used as default values for the configs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
