emanuilov commented on PR #3165: URL: https://github.com/apache/iceberg-rust/pull/3165#issuecomment-6574417090
Related: [#3605](https://github.com/apache/iceberg-rust/pull/3605) makes the S3 retry policy configurable, and it would be good for this backend to use the same settings. Why: S3, S3 Tables included, answers `503 SlowDown` while it scales a prefix up. High-throughput writers hit this routinely, for example a backfill flushing hundreds of partition files at once, or many writers creating files in new tables in one table bucket. AWS's guidance is to retry with exponential backoff and jitter until S3 has scaled. Today the opendal backend hard-codes `RetryLayer::new()` (3 retries starting at 1 s, no jitter), so the error reaches the writer after about 7 s and the whole commit fails. This backend uses `object_store`'s default `RetryConfig`, which isn't configurable from Iceberg properties either. What [#3605](https://github.com/apache/iceberg-rust/pull/3605) does: it adds Iceberg Java's existing S3FileIO keys to `iceberg::io`, with the same names and meaning, so one catalog config behaves the same from Java and Rust: - `s3.retry.num-retries` (Java's default is 5; its docs recommend 32 for high-throughput workloads) - `s3.retry.min-wait-ms` - `s3.retry.max-wait-ms` The opendal S3 operator applies them to its `RetryLayer`, and setting any of them turns jitter on. With none set, nothing changes. It's small and self-contained (the keys, parsing and unit tests). Suggestion: if [#3605](https://github.com/apache/iceberg-rust/pull/3605) merges first, this PR could rebase on it and read the same keys through `AmazonS3Builder::with_retry`: - `s3.retry.num-retries` → `RetryConfig::max_retries` - `s3.retry.min-wait-ms` → `BackoffConfig::init_backoff` - `s3.retry.max-wait-ms` → `BackoffConfig::max_backoff` With none set, `object_store`'s defaults would stay as they are. Both S3 backends would then share one retry configuration. Happy to help with that part if useful. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
