Yida Wu has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/22930 )
Change subject: WIP: IMPALA-6406: Add CatalogActionLocker for fine grained DDL locking ...................................................................... WIP: IMPALA-6406: Add CatalogActionLocker for fine grained DDL locking Replace global metadata DDL lock with CatalogActionLocker for fine grained locking of DDL operations. The previous global lock could block the entire catalog server during slow HMS operations. We still keep the existing table and database locks. However, this new locker focuses on the whole operation, it grabs the needed locks at the beginning and holds them until the operation finishes. The new locker uses database and table level locks, allowing DDL operations to run at the same time when possible, using shared and exclusive locks. This helps reduce blocking and improves performance. In the future, it may also help reject duplicate actions early without long waiting (not included in this patch). Right now, only the operations that used the global DDL lock are updated to use CatalogActionLocker. We may use it for more operations later. Tests: Adds CatalogActionLockerTest for concurrency and correctness check. Passed custom_cluster/test_concurrent_ddls.py. TODO more tests. Change-Id: I2ed48d92333787f75b64b8bd878834612433f6f8 --- A fe/src/main/java/org/apache/impala/catalog/CatalogActionLocker.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/metastore/CatalogMetastoreServiceHandler.java M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java A fe/src/test/java/org/apache/impala/catalog/CatalogActionLockerTest.java 6 files changed, 1,017 insertions(+), 71 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/30/22930/4 -- To view, visit http://gerrit.cloudera.org:8080/22930 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2ed48d92333787f75b64b8bd878834612433f6f8 Gerrit-Change-Number: 22930 Gerrit-PatchSet: 4 Gerrit-Owner: Yida Wu <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Michael Smith <[email protected]>
