ChristinaTech commented on issue #7151: URL: https://github.com/apache/iceberg/issues/7151#issuecomment-1477156740
Yeah, having dug into a bit more and gone over https://github.com/apache/iceberg/pull/3717 since I wrote that, I'm not certain if Option 5 is the best solution anymore either as it turns a lot of cases where we really should know if we failed into UNKNOWN cases. However, I am uncertain whether we want to tell the Glue client not to retry after a timeout or 500 error in all cases, as that could hurt the client's robustness in other scenarios. If we are gonna give it a custom retry condition anyways, it would make more sense to create an [ExecutionAttribute](https://github.com/aws/aws-sdk-java-v2/blob/master/core/sdk-core/src/main/java/software/amazon/awssdk/core/interceptor/ExecutionAttribute.java) we can attach to any individual API call we want not to retry, create a `retryCondition` that checks for it, and then build Option 3 using Iceberg Tasks as I described it above instead to avoid more side effects than necessary. The `ExecutionAttribute` and custom `retryCondition` can also be replaced with a second Glue Client, but that will require changes to the AWS Client factory interface, which is a public interface, so would rather avoid that. As a secondary concern, right now, we actually treat a 500 error as a delete-the-metadata type FAILURE when the commit status after that could technically be SUCCESS or UNKNOWN. For example, its completely possible that Glue (or DynamoDB) could succeed on the backend and then error out during some post-commit step or even response marshalling. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
