[
https://issues.apache.org/jira/browse/HADOOP-18759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anuj Modi updated HADOOP-18759:
-------------------------------
Description:
Today when a request fails with connection timeout, it falls back into the loop
for exponential retry. Unlike Azure Storage, there are no guarantees of success
on exponentially retried request or recommendations for ideal retry policies
for Azure network or any other general failures. Faster failure and retry might
be more beneficial for such generic connection timeout failures.
This PR introduces a new Linear Retry Policy which will currently be used only
for Connection Timeout failures.
Two types of Linear Backoff calculations will be supported:
# min-backoff starts with 500 ms and with each attempted retry, back-off
increments double, capped at 30 sec max
# min-backoff starts with 500 ms and with each attempted retry, back-off
increments by 1 sec, capped at 30 sec max
Following Configurations will be introduced in the change:
1. {*}"fs.azure.linear.retry.for.connection.timeout.enabled{*}" -
{*}default{*}: true, {*}true{*}: linear retry will be used for CT, {*}false{*}:
Exponential retry will be used.
2. "{*}fs.azure.io.retry.min.backoff.interval.for.connection.timeout{*}" -
{*}default{*}: 500ms
3. "{*}fs.azure.io.retry.max.backoff.interval.for.connection.timeout{*}" -
{*}default{*}: 30s
4. {*}"fs.azure.linear.retry.double.step.up.enabled{*}" - {*}default{*}: true,
{*}true{*}: Double up the interval for every retry count, {*}false{*}: Add 1s
to interval for every retry count
was:
Today when a request fails with connection timeout, it falls back into the loop
for exponential retry. Unlike Azure Storage, there are no guarantees of success
on exponentially retried request or recommendations for ideal retry policies
for Azure network or any other general failures. Faster failure and retry might
be more beneficial for such generic connection timeout failures.
This PR introduces a new Linear Retry Policy which will currently be used only
for Connection Timeout failures.
Two types of Linear Backoff calculations will be supported:
# min-backoff starts with 500 ms and with each attempted retry, back-off
increments double, capped at 30 sec max
# min-backoff starts with 500 ms and with each attempted retry, back-off
increments by 1 sec, capped at 30 sec max
> [ABFS][Backoff-Optimization] Have a Linear retry policy for connection
> timeout failures
> ---------------------------------------------------------------------------------------
>
> Key: HADOOP-18759
> URL: https://issues.apache.org/jira/browse/HADOOP-18759
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
> Affects Versions: 3.3.4
> Reporter: Anuj Modi
> Assignee: Anuj Modi
> Priority: Major
>
> Today when a request fails with connection timeout, it falls back into the
> loop for exponential retry. Unlike Azure Storage, there are no guarantees of
> success on exponentially retried request or recommendations for ideal retry
> policies for Azure network or any other general failures. Faster failure and
> retry might be more beneficial for such generic connection timeout failures.
> This PR introduces a new Linear Retry Policy which will currently be used
> only for Connection Timeout failures.
> Two types of Linear Backoff calculations will be supported:
> # min-backoff starts with 500 ms and with each attempted retry, back-off
> increments double, capped at 30 sec max
> # min-backoff starts with 500 ms and with each attempted retry, back-off
> increments by 1 sec, capped at 30 sec max
> Following Configurations will be introduced in the change:
> 1. {*}"fs.azure.linear.retry.for.connection.timeout.enabled{*}" -
> {*}default{*}: true, {*}true{*}: linear retry will be used for CT,
> {*}false{*}: Exponential retry will be used.
> 2. "{*}fs.azure.io.retry.min.backoff.interval.for.connection.timeout{*}" -
> {*}default{*}: 500ms
> 3. "{*}fs.azure.io.retry.max.backoff.interval.for.connection.timeout{*}" -
> {*}default{*}: 30s
> 4. {*}"fs.azure.linear.retry.double.step.up.enabled{*}" - {*}default{*}:
> true, {*}true{*}: Double up the interval for every retry count, {*}false{*}:
> Add 1s to interval for every retry count
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
