Hello Bharath Vissapragada, Tianyi Wang, Vuk Ercegovac,
I'd like you to do a code review. Please visit
http://gerrit.cloudera.org:8080/11280
to review the following change.
Change subject: WIP: IMPALA-7469. Invalidate LocalCatalog cache based on topic
updates
......................................................................
WIP: IMPALA-7469. Invalidate LocalCatalog cache based on topic updates
This implements cache invalidation inside CatalogdMetaProvider. The
design is as follows:
- when the catalogd collects updates into the statestore topic, it now
adds an additional entry for each table and database. These additional
entries are minimal - they only include the object's name, but no
metadata.
- the old-style topic entries are prefixed with a '1:' whereas the new
minimal entries are prefixed with a '2:'. The impalad will subscribe
to one or the other prefix depending on whether it is running with
--use_local_catalog. Thus, old impalads will not be confused by the
new entries and vice versa.
- when the impalad gets these topic updates, it forwards them through to
the catalog implementation. The LocalCatalog implementation forwards
them to the CatalogdMetaProvider, which uses them to invalidate
cached metadata as appropriate.
This patch includes some basic unit tests. I also did some manual
testing by connecting to different impalads and verifying that a session
connected to impalad #1 saw the effects of DDLs made by impalad #2
within a short period of time (the statestore topic update frequency).
This patch is marked as a WIP since I'd like to evaluate the current
e2e test coverage and see if we need any new specific tests for
cross-impalad consistency.
Change-Id: I615f9e6bd167b36cd8d93da59426dd6813ae4984
---
M be/src/service/impala-server.cc
M common/thrift/CatalogObjects.thrift
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java
M fe/src/main/java/org/apache/impala/common/Pair.java
M fe/src/main/java/org/apache/impala/service/FeCatalogManager.java
M fe/src/test/java/org/apache/impala/catalog/local/CatalogdMetaProviderTest.java
M fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java
9 files changed, 404 insertions(+), 67 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/80/11280/1
--
To view, visit http://gerrit.cloudera.org:8080/11280
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I615f9e6bd167b36cd8d93da59426dd6813ae4984
Gerrit-Change-Number: 11280
Gerrit-PatchSet: 1
Gerrit-Owner: Todd Lipcon <[email protected]>
Gerrit-Reviewer: Bharath Vissapragada <[email protected]>
Gerrit-Reviewer: Tianyi Wang <[email protected]>
Gerrit-Reviewer: Vuk Ercegovac <[email protected]>
