Yida Wu has uploaded this change for review. ( http://gerrit.cloudera.org:8080/22930
Change subject: WIP: IMPALA-6406: Add CatalogActionLocker for fine grained DDL locking ...................................................................... WIP: IMPALA-6406: Add CatalogActionLocker for fine grained DDL locking Replace global metadata DDL lock with CatalogActionLocker for fine grained locking of DDL operations. The previous global lock could block the entire catalog server during slow HMS operations. We still keep the existing table and database locks. However, this new locker focuses on the whole operation, it grabs the needed locks at the beginning and holds them until the operation finishes. The new locker uses database and table level locks, allowing DDL operations to run at the same time when possible, using shared and exclusive locks. This helps reduce blocking and improves performance. This locker grabs locks early, which may hold them longer than late locking but should be good enough for ddls like CREATE_DB or DROP_TABLE with database and table level locks, especially compared to the current global locking. It could simplify correctness and the centralized structure makes it easier to see which operation holds which locks, improving debugging. In the future, it may also help reject duplicate actions early without long waiting (not included in this patch). Right now, only the operations that used the global metadata DDL lock are updated to use CatalogActionLocker. We may use it for more operations later. Tests: Adds CatalogActionLockerTest for concurrency and correctness check. Passed custom_cluster/test_concurrent_ddls.py. TODO more tests. Change-Id: I2ed48d92333787f75b64b8bd878834612433f6f8 --- A fe/src/main/java/org/apache/impala/catalog/CatalogActionLocker.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/metastore/CatalogMetastoreServiceHandler.java M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java A fe/src/test/java/org/apache/impala/catalog/CatalogActionLockerTest.java 6 files changed, 904 insertions(+), 71 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/30/22930/1 -- To view, visit http://gerrit.cloudera.org:8080/22930 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I2ed48d92333787f75b64b8bd878834612433f6f8 Gerrit-Change-Number: 22930 Gerrit-PatchSet: 1 Gerrit-Owner: Yida Wu <[email protected]>
