borislitvak commented on issue #2330: URL: https://github.com/apache/hudi/issues/2330#issuecomment-743375285
@vinothchandar I have never worked with Hive, but: - It seems that the Hive (Metastore?) locking is done in Zookeeper. Thus I am not sure Iceberg is taking the logs there. - This locking can be [per partition](https://cwiki.apache.org/confluence/display/Hive/Locking) Note that I have not encountered any locking API for [AWS Glue Catalog](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-catalog-tables.html#aws-glue-api-catalog-tables-actions), which further supports the Zookeeper hypothesis. _Suggestion 1_: If only a partition can be locked, it can allow for table designs that mitigate the global lock issue you've described above. This should take much less time than implementing CRDT, which bringing benefit. The downside will be partition based consistency ala [delta](https://docs.delta.io/0.7.0/presto-integration.html). _Suggestion 2_ While on this topic, with read-after-write S3 guarantees, you can just keep this "lock" in S3, IMO. If this does not take any sense, that's OK, please state so. Again, I have no experience with Hive. I don't have the time to dive any deeper. Hope this helps, Boris. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
