Dimitris Tsirogiannis has uploaded this change for review. ( http://gerrit.cloudera.org:8080/8545
Change subject: [PREVIEW] IMPALA-5058: Improve concurrency of DDL/DML operations ...................................................................... [PREVIEW] IMPALA-5058: Improve concurrency of DDL/DML operations Problem: A long running table metadata operation (e.g. refresh) could prevent any other metadata operation from making progress if it coincided with the gather catalog updates operation. The problem was due to the conservative locking scheme used when catalog updates were collected. In particular, in order to collect a consistent snapshot of metadata changes, the global catalog lock was held for the entire duration of that operation. Solution: To improve the concurrency of catalog operations the following changes are performed: * A range of catalog versions determines the catalog changes to be included in a catalog update. Any catalog changes that do not fall in the specified range are ignored (to be processed in subsequent catalog updates). * The catalog allows metadata operations to make progress while collecting catalog updates. * To prevent starvation of catalog updates (i.e. frequently updated catalog objects skipping catalog updates indefinitely), we keep track of the number of times a catalog object has skipped an update and if that number exceeds a threshold it is included in the next catalog update even if its version is not in the specified catalog update version range. Hence, the same catalog object may be sent in two consecutive catalog updates. Change-Id: I3032437f83d39bcc8cff14d897c7c106a4ab62d3 --- M be/src/exec/catalog-op-executor.cc M be/src/exec/catalog-op-executor.h M be/src/service/client-request-state.cc M be/src/service/frontend.cc M be/src/service/impala-server.cc M be/src/service/impala-server.h M common/thrift/CatalogService.thrift M common/thrift/Frontend.thrift M fe/src/main/java/org/apache/impala/catalog/AuthorizationPolicy.java M fe/src/main/java/org/apache/impala/catalog/Catalog.java M fe/src/main/java/org/apache/impala/catalog/CatalogDeltaLog.java M fe/src/main/java/org/apache/impala/catalog/CatalogObject.java M fe/src/main/java/org/apache/impala/catalog/CatalogObjectCache.java A fe/src/main/java/org/apache/impala/catalog/CatalogObjectImpl.java A fe/src/main/java/org/apache/impala/catalog/CatalogObjectVersionManager.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/DataSource.java M fe/src/main/java/org/apache/impala/catalog/Db.java M fe/src/main/java/org/apache/impala/catalog/Function.java M fe/src/main/java/org/apache/impala/catalog/HdfsCachePool.java M fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java M fe/src/main/java/org/apache/impala/catalog/Role.java M fe/src/main/java/org/apache/impala/catalog/RolePrivilege.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java M fe/src/main/java/org/apache/impala/util/SentryProxy.java M fe/src/test/java/org/apache/impala/testutil/CatalogServiceTestCatalog.java M fe/src/test/java/org/apache/impala/testutil/ImpaladTestCatalog.java 30 files changed, 1,153 insertions(+), 458 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/45/8545/1 -- To view, visit http://gerrit.cloudera.org:8080/8545 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I3032437f83d39bcc8cff14d897c7c106a4ab62d3 Gerrit-Change-Number: 8545 Gerrit-PatchSet: 1 Gerrit-Owner: Dimitris Tsirogiannis <[email protected]>
