rdblue commented on PR #5688: URL: https://github.com/apache/iceberg/pull/5688#issuecomment-1235950628
@gaborkaszab, @szehon-ho, I think there's a better way than controlling the number of retries. I doubt anyone is going to actually set this until after they are burned by it. I think the solution is to fail if the retry is because the file doesn't exist. I opened a draft PR here: https://github.com/apache/iceberg/pull/5696 The only problem with that is how to handle cases where you want to retry because of eventual consistency. However, we already handle eventual consistency using read-after-write consistency. To ensure we get read-after-write consistency in S3, we always call `overwrite` since we know metadata locations are unique (contain a random UUID). See https://github.com/apache/iceberg/blob/master/core/src/main/java/org/apache/iceberg/BaseMetastoreTableOperations.java#L160-L162 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
