Dimitris Tsirogiannis has uploaded this change for review. (
http://gerrit.cloudera.org:8080/8266
Change subject: IMPALA-5058: Improve concurrency of DDL/DML during catalog
updates
..
IMPALA-5058: Improve concurrency of DDL/DML during catalog updates
Problem: A long running table metadata operation (e.g. refresh) could
prevent any other metadata operation from making progress if it
coincided with the gather catalog updates operation. The problem was due
to the conservative locking scheme used when catalog updates were
collected. In particular, in order to collect a consistent snapshot of
metadata changes, the global catalog lock was held for the entire
duration of that operation.
Solution: To improve the concurrency of catalog operations the following
changes are performed:
* A range of catalog versions determines the catalog changes to be
included in a catalog update. Any catalog changes that do not fall in
the specified range are ignored (to be processed in subsequent catalog
updates).
* The catalog allows metadata operations to make progress while collecting
catalog updates.
* To prevent starvation of catalog updates (i.e. frequently updated
catalog objects skipping catalog updates indefinitely), we keep track
of the number of times a catalog object has skipped an update and if
that number exceeds a threshold it is included in the next catalog
update even if its version is not in the specified catalog update
version range. Hence, the same catalog object may be sent in two
consecutive catalog updates.
Change-Id: I16ae27be79eb0c5a9648a15d368ca60a7d04507b
---
M fe/src/main/java/org/apache/impala/catalog/CatalogDeltaLog.java
M fe/src/main/java/org/apache/impala/catalog/CatalogObject.java
A fe/src/main/java/org/apache/impala/catalog/CatalogObjectImpl.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/DataSource.java
M fe/src/main/java/org/apache/impala/catalog/Db.java
M fe/src/main/java/org/apache/impala/catalog/Function.java
M fe/src/main/java/org/apache/impala/catalog/HdfsCachePool.java
M fe/src/main/java/org/apache/impala/catalog/Role.java
M fe/src/main/java/org/apache/impala/catalog/RolePrivilege.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
11 files changed, 265 insertions(+), 191 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/66/8266/1
--
To view, visit http://gerrit.cloudera.org:8080/8266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I16ae27be79eb0c5a9648a15d368ca60a7d04507b
Gerrit-Change-Number: 8266
Gerrit-PatchSet: 1
Gerrit-Owner: Dimitris Tsirogiannis