rdblue commented on pull request #3663: URL: https://github.com/apache/iceberg/pull/3663#issuecomment-986291413
@CodingCat, `HadoopTableOperations` is not safe to use with a file system that doesn't support atomic rename. Even if we were to add a lock as proposed in this PR, concurrent processes could still produce the same version in stores like S3 or even most local file systems. If you want to avoid using a catalog implementation that provides an atomic rename (like JDBC, DynamoDB, or Hive) then you'd need to use a lock that can coordinate across processes. For using Glue as a metastore, @jackye1995 added a DynamoDB based lock, in `DynamoLockManager`. I recommend adding the ability for `HadoopTableOperations` to use the DynamoDB lock instead of adding a JVM lock. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
