ookumuso commented on code in PR #11052:
URL: https://github.com/apache/iceberg/pull/11052#discussion_r1764233622
##########
aws/src/main/java/org/apache/iceberg/aws/s3/S3FileIOProperties.java:
##########
@@ -393,6 +403,21 @@ public class S3FileIOProperties implements Serializable {
*/
private static final String S3_FILE_IO_USER_AGENT = "s3fileio/" +
EnvironmentContext.get();
+ /** Number of times to retry S3 operations. */
+ public static final String S3_RETRY_NUM_RETRIES = "s3.retry.num-retries";
+
+ public static final int S3_RETRY_NUM_RETRIES_DEFAULT = 32;
Review Comment:
@amogh-jahagirdar Main consideration here is to give S3 enough time to
auto-scale particularly for workloads which observe 503s when they go beyond S3
advertised limits. What we observed in our load tests that high concurrent
workloads exhaust the default retries(3) and fail almost immediately. How fast
these retries exhausted will depend on the concurrency level. Retry count makes
the main difference here since the overall application becomes more resistant
to 503s which leads to success and better tps scaling for upcoming workloads
with auto-scaling.
I would say in average case particularly workloads running below advertised
limits, customers will not see a change in behavior since default retries
wouldn't be exhausted for those anyway but as we see more adoption to Iceberg
with bigger workloads running; exhausting retries on throttles becomes an
issue. One downside would be that if there is a more systemic error in
communications with S3, workloads will take longer time to fail, up to 10
minutes but I would argue that those will be more on the rare side compared to
the value it brings for eventually succeeding workloads.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
