kbendick commented on pull request #2552: URL: https://github.com/apache/iceberg/pull/2552#issuecomment-833238321
> I think this is a good fix to the NPE. But I do want to make sure we warn folks that an eventually consistent Catalog can break the history of an Iceberg table. This would only be acceptable for a catalog which guaranteed consistent reads during the lock phase. > > My thinking is basically > > ``` > Consistent - Read Your Writes Catalog = Best > Eventually Consistent Reads - Consistent when locking/committing = Adequate but not great > Eventually consistent during lock/commit = Will cause lost writes, non-linearizable histories and weirdness. > ``` Is this something that could come up with the DynamoDB based lock? I agree this is a good fix to the NPE, but I have personally had weird edge cases with other software that uses a DynamoDB table as a lock (such as terraform). Ideally testing handles this and the DynamoDB / Glue catalog doesn't suffer from these issues. Possibly I need to review the DynamoDB table lock again, but felt it might be worth noting as I've had weird issues with terraform table locks and consistency issues. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
