rtyler commented on PR #513: URL: https://github.com/apache/arrow-rs-object-store/pull/513#issuecomment-3419805934
The graph [linked](https://github.com/apache/arrow-rs-object-store/issues/51#issuecomment-2741692035) by @westonpace reflects what we have been seeing in real-world scenarios. I think the default settings for backoff are non-functional. With the AmazonS3 object store we regularly see operations fail with only about 1s of "total retries" since each backoff step is roughly 100ms. In a lot of ways this is not fundamentally different than just a `loop {}` on the operation. The potential retry loop executing over 400 times before the max retries is exhausted I don't think is reasonable for any real-world usage. This behavior is particularly problematic when the retries are occurring due to a rate-limiting situation, `object_store` basically makes matters _dramatically_ worse with its current retry and backoff implementation -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
