kasakrisz commented on PR #3934: URL: https://github.com/apache/hive/pull/3934#issuecomment-1387079817
Lock requests in the code: 1. https://github.com/apache/hive/blob/55471330426c2e0a52101c2e535a66f751be76ee/ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java#L3029 2. https://github.com/apache/hive/blob/55471330426c2e0a52101c2e535a66f751be76ee/iceberg/iceberg-catalog/src/main/java/org/apache/iceberg/hive/HiveCommitLock.java#L118 The first lock is acquired only if these setting are present ``` set hive.support.concurrency=true; set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager; ``` These are required for native acid operations. The deadlock happens in a few lines later in the 2. place: after the lock request is issued we check periodically the state of the lock whether it is acquired or not. Unfortunately in this use case it is always in a waiting state since another exclusive write lock already acquired on the same table in the 1. place and that one is released only when the current txn ends. In this patch my goal was to copy the behavior from the insert case: let the table's storage handler determine the type of lock in the 1. place. https://github.com/apache/hive/blob/55471330426c2e0a52101c2e535a66f751be76ee/ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java#L3068 In case of Iceberg it is `SHARED_READ`. Iceberg uses optimistic concurrency control so multiple transactions can write the same table parallel only the Iceberg commit step is serialized using exclusive write locks. Currently we support only full rebuild of Iceberg Materialized views which is technically an insert overwrite. So a plain insert overwrite statement is also affected. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
