Hello Bharath Vissapragada, Vihang Karajgaonkar, Todd Lipcon, Impala Public
Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/14307
to look at the new patch set (#6).
Change subject: IMPALA-7506: support global INVALIDATE METADATA in local
catalog mode
......................................................................
IMPALA-7506: support global INVALIDATE METADATA in local catalog mode
The minimal catalog object version of valid catalog objects is used to
implement global invalidate metadata in legacy catalog mode. Coordinator
sends DDL RPC to catalogd for global invalidate metadata and gets the
expected min catalog version in the response. It's the version when
catalogd starts to reset the entire catalog, which means when the reset
is done, all valid catalog objects should be associated with a catalog
version larger than it. Coordinator will wait until its min catalog
version exceeds this value, which means it has processed all the updates
of the reset propagated from the catalogd via statestored. If SYNC_DDL
is set, the coordinator will also wait until other coordinators reach
the same statestore topic version with it, so they have also processed
the same updates and had the latest catalog after reset.
In local catalog mode, the coordinator does not cache all the metadata.
Instead, it caches them on-demand (based on query requests), and removes
them based on the Guava cache configurations (size or TTL) or explicit
invalidation from the catalog topic updates. So it's hard to track the
minimal catalog object version correctly.
This patch adds a new field (lastResetCatalogVersion) in TCatalog to
propagate the catalog version when catalogd starts to reset the entire
metadata. Each time when catalogd generates a new topic update, it will
generate a TCatalogObject of CATALOG type containing the state of the
catalog which includes this new field.
To make all changes of the reset being added in the same topic update
with this TCatalog object. Rapidly changed tables that have catalog
version exceeding the version range of this update will also be included.
When coordinator receives a new value of lastResetCatalogVersion in a
topic update, it means catalogd has reset the entire catalog and all the
relative updates are whether included in the same or previous topic
updates. This is guaranteed by three facts:
1) No topic updates are sent from catalogd when the write lock of
versionLock is held in CatalogServiceCatalog.reset(). Note that the
update thread requires holding the read lock of versionLock.
2) Authz changes before holding the write lock can only be sent in a
previous topic update or in the next topic update after reset().
3) No catalog objects are skipped in the topic update right after
reset(). See changes in GetCatalogDeltaContext.
Thus, all metadata with catalog version <= lastResetCatalogVersion can be
considered stale after coordinator finish processing the topic update.
lastResetCatalogVersion + 1 is the lower bound (included) of min catalog
object version of a coordinator.
To avoid catalogd's update collector thread being blocked by concurrent
DDLs that holding the table locks, this patch also fixes IMPALA-9062. In
local catalog mode, we just need to propagate the table name of a
changed table, so don't need to acquire table lock to get a full TTable
object.
This patch also exposes the min catalog object version of coordinator
via a new metric "catalog.min-catalog-object-version" to ease debugging.
Tests:
- Recover all existing tests that have been disabled due to this
missing feature
Change-Id: Ib61a7ab1ffa062620ffbc2dadc34bd7a8ca9e549
---
M be/src/service/impala-server.cc
M be/src/util/impalad-metrics.cc
M be/src/util/impalad-metrics.h
M common/thrift/CatalogObjects.thrift
M common/thrift/metrics.json
M fe/src/main/java/org/apache/impala/analysis/ResetMetadataStmt.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M tests/authorization/test_grant_revoke.py
M tests/authorization/test_ranger.py
M tests/common/skip.py
M tests/custom_cluster/test_local_catalog.py
M tests/metadata/test_hms_integration.py
M tests/metadata/test_metadata_query_statements.py
14 files changed, 180 insertions(+), 106 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/07/14307/6
--
To view, visit http://gerrit.cloudera.org:8080/14307
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib61a7ab1ffa062620ffbc2dadc34bd7a8ca9e549
Gerrit-Change-Number: 14307
Gerrit-PatchSet: 6
Gerrit-Owner: Quanlong Huang <huangquanl...@gmail.com>
Gerrit-Reviewer: Bharath Vissapragada <bhara...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <huangquanl...@gmail.com>
Gerrit-Reviewer: Todd Lipcon <t...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vih...@cloudera.com>
