emanuilov commented on code in PR #3605: URL: https://github.com/apache/iceberg-rust/pull/3605#discussion_r4446872728
########## crates/iceberg/src/io/storage/config/s3.rs: ########## @@ -65,6 +65,13 @@ pub const S3_ALLOW_ANONYMOUS: &str = "s3.allow-anonymous"; pub const S3_DISABLE_EC2_METADATA: &str = "s3.disable-ec2-metadata"; /// Option to skip loading configuration from config file and the env. pub const S3_DISABLE_CONFIG_LOAD: &str = "s3.disable-config-load"; +/// Number of times to retry an S3 operation, `0` to disable retrying. Retries back off exponentially, +/// with jitter, between [`S3_RETRY_MIN_WAIT_MS`] and [`S3_RETRY_MAX_WAIT_MS`]. +pub const S3_RETRY_NUM_RETRIES: &str = "s3.retry.num-retries"; +/// Minimum wait in milliseconds before retrying an S3 operation. +pub const S3_RETRY_MIN_WAIT_MS: &str = "s3.retry.min-wait-ms"; +/// Maximum wait in milliseconds between two retries of an S3 operation. +pub const S3_RETRY_MAX_WAIT_MS: &str = "s3.retry.max-wait-ms"; Review Comment: Pushed the change with the layered config: opendal.retry.* for every backend, with s3.retry.* overriding per key for S3. Tested locally: - The four existing retry tests are unchanged. New tests cover generic keys applying to every backend, S3 keys winning per key over the generic ones, a key set at only one layer, and the new keys rejecting invalid values. - Downstream: I built our Kafka → Iceberg (S3 Tables) sink against this branch (iceberg, iceberg-catalog-s3tables, iceberg-storage-opendal patched to it) and ran its end-to-end tests on a local kind cluster against real S3 Tables: - with s3.retry.num-retries=10, s3.retry.min-wait-ms=2000, s3.retry.max-wait-ms=20000 and opendal.retry.num-retries=3 all set, the run committed every row exactly once; - with an invalid opendal.retry.min-wait-ms=0, and separately s3.retry.min-wait-ms=0, the sink fails at startup with a property error naming the key, so both layers are read through the real FileIO path. I didn’t reproduce real S3 throttling here, so which layer wins at runtime is covered by the unit tests rather than the e2e. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
