[
https://issues.apache.org/jira/browse/HADOOP-19096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17833235#comment-17833235
]
ASF GitHub Bot commented on HADOOP-19096:
-----------------------------------------
anujmodi2021 commented on code in PR #6276:
URL: https://github.com/apache/hadoop/pull/6276#discussion_r1548144570
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsRestOperation.java:
##########
@@ -321,7 +323,27 @@ private boolean executeHttpOperation(final int retryCount,
} else if (httpOperation.getStatusCode() ==
HttpURLConnection.HTTP_UNAVAILABLE) {
incrementCounter(AbfsStatistic.SERVER_UNAVAILABLE, 1);
}
+
+ // If no exception occurred till here it means http operation was
successfully complete and
+ // a response from server has been received which might be failure or
success.
+ // If any kind of exception has occurred it will be caught below.
+ // If request failed determine failure reason and retry policy here.
+ // else simply return with success after saving the result.
+ LOG.debug("HttpRequest: {}: {}", operationType, httpOperation);
+
+ int status = httpOperation.getStatusCode();
+ failureReason = RetryReason.getAbbreviation(null, status,
httpOperation.getStorageErrorMessage());
+ retryPolicy = client.getRetryPolicy(failureReason);
+
+ if (retryPolicy.shouldRetry(retryCount, httpOperation.getStatusCode())) {
+ return false;
+ }
+
+ // If the request has succeeded or failed with non-retrial error, save
the operation and return.
+ result = httpOperation;
+
} catch (UnknownHostException ex) {
+ wasExceptionThrown = true;
Review Comment:
Changed it to wasKnownExceptionThrown.
So that if any other exception is thrown we will still update metrics
> [ABFS] Enhancing Client-Side Throttling Metrics Updation Logic
> --------------------------------------------------------------
>
> Key: HADOOP-19096
> URL: https://issues.apache.org/jira/browse/HADOOP-19096
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
> Affects Versions: 3.4.1
> Reporter: Anuj Modi
> Assignee: Anuj Modi
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.1
>
>
> ABFS has a client-side throttling mechanism which works on the metrics
> collected from past requests made. I requests are getting failed due to
> throttling at server, we update our metrics and client side backoff is
> calculated based on those metrics.
> This PR enhances the logic to decide which requests should be considered to
> compute client side backoff interval as follows:
> For each request made by ABFS driver, we will determine if they should
> contribute to Client-Side Throttling based on the status code and result:
> # Status code in 2xx range: Successful Operations should contribute.
> # Status code in 3xx range: Redirection Operations should not contribute.
> # Status code in 4xx range: User Errors should not contribute.
> # Status code is 503: Throttling Error should contribute only if they are
> due to client limits breach as follows:
> ## 503, Ingress Over Account Limit: Should Contribute
> ## 503, Egress Over Account Limit: Should Contribute
> ## 503, TPS Over Account Limit: Should Contribute
> ## 503, Other Server Throttling: Should not Contribute.
> # Status code in 5xx range other than 503: Should not Contribute.
> # IOException and UnknownHostExceptions: Should not Contribute.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
