Hello Bharath Vissapragada, Vihang Karajgaonkar, Todd Lipcon, Impala Public
Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/14307
to look at the new patch set (#12).
Change subject: IMPALA-7506: support global INVALIDATE METADATA in local
catalog mode
......................................................................
IMPALA-7506: support global INVALIDATE METADATA in local catalog mode
The minimal catalog object version of valid catalog objects is used to
implement global invalidate metadata in legacy catalog mode. Coordinator
sends DDL RPC to catalogd for global invalidate metadata and gets the
expected min catalog version in the response. It's the version when
catalogd starts to reset the entire catalog, which means when the reset
is done, all valid catalog objects should be associated with a catalog
version larger than it. Coordinator will wait until its min catalog
version exceeds this value, which means it has processed all the updates
of the reset propagated from the catalogd via statestored. If SYNC_DDL
is set, the coordinator will also wait until other coordinators reach
the same statestore topic version with it, so they have also processed
the same updates and had the latest catalog after reset.
In local catalog mode, the coordinator does not cache all the metadata.
Instead, it caches them on-demand (based on query requests), and removes
them based on the Guava cache configurations (size or TTL) or explicit
invalidation from the catalog topic updates. So it's hard to track the
minimal catalog object version correctly.
This patch adds a new field (lastResetCatalogVersion) in TCatalog to
propagate the catalog version when catalogd starts to reset the entire
metadata. Each time when catalogd generates a new topic update, it will
generate a TCatalogObject of CATALOG type containing the state of the
catalog which includes this new field.
When coordinator receives a new value of lastResetCatalogVersion in a
topic update, it means catalogd has reset the entire catalog.
Coordinator will then clear its cache to remove all stale catalog
objects. It's possible that some fresh items being removed too. They
will be refetched on demand.
After the invalidation, there are no catalog object cached with catalog
version <= lastResetCatalogVersion. Because stale cache has been cleared
and all metadata from catalogd is newer than lastResetCatalogVersion. So
lastResetCatalogVersion + 1 is the lower bound (included) of min catalog
object version of a coordinator.
This patch also exposes the lower bound of catalog object version of via
a new metric "catalog.catalog-object-version-lower-bound" to ease
debugging.
IMPALA-9136 is also fixed in this patch.
Tests:
- Recover all existing tests that have been disabled due to this
missing feature
- Add custom cluster test for concurrent DDLs with INVALIDATE METADATA
- Run CORE tests
Change-Id: Ib61a7ab1ffa062620ffbc2dadc34bd7a8ca9e549
---
M be/src/service/impala-server.cc
M be/src/service/impala-server.h
M be/src/util/impalad-metrics.cc
M be/src/util/impalad-metrics.h
M common/thrift/CatalogObjects.thrift
M common/thrift/Frontend.thrift
M common/thrift/metrics.json
M fe/src/main/java/org/apache/impala/analysis/ResetMetadataStmt.java
M fe/src/main/java/org/apache/impala/catalog/CatalogObjectImpl.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M tests/authorization/test_grant_revoke.py
M tests/authorization/test_ranger.py
M tests/common/skip.py
A tests/custom_cluster/test_concurrent_ddls.py
M tests/custom_cluster/test_local_catalog.py
M tests/metadata/test_hms_integration.py
M tests/metadata/test_metadata_query_statements.py
19 files changed, 385 insertions(+), 117 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/07/14307/12
--
To view, visit http://gerrit.cloudera.org:8080/14307
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib61a7ab1ffa062620ffbc2dadc34bd7a8ca9e549
Gerrit-Change-Number: 14307
Gerrit-PatchSet: 12
Gerrit-Owner: Quanlong Huang <huangquanl...@gmail.com>
Gerrit-Reviewer: Bharath Vissapragada <bhara...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <huangquanl...@gmail.com>
Gerrit-Reviewer: Todd Lipcon <t...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vih...@cloudera.com>
