Sneha Varma created HADOOP-15703:
------------------------------------
Summary: AzureBlobFS - implement client-side throttling
Key: HADOOP-15703
URL: https://issues.apache.org/jira/browse/HADOOP-15703
Project: Hadoop Common
Issue Type: Sub-task
Reporter: Sneha Varma
Big data workloads frequently exceed the AzureBlobFS max ingress and egress
limits
(https://docs.microsoft.com/en-us/azure/storage/common/storage-scalability-targets).
For example, the max ingress limit for a GRS account in the United States is
currently 10 Gbps. When the limit is exceeded, the AzureBlobFS service fails a
percentage of incoming requests, and this causes the client to initiate the
retry policy. The retry policy delays requests by sleeping, but the sleep
duration is independent of the client throughput and account limit. This
results in low throughput, due to the high number of failed requests and
thrashing causes by the retry policy.
To fix this, we introduce a client-side throttle which minimizes failed
requests and maximizes throughput.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
