Hello Tidy Bot, Kudu Jenkins, Adar Dembo, Hao Hao,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/8313
to look at the new patch set (#33).
Change subject: KUDU-2191 (12/n): Hive Metastore notification log event listener
......................................................................
KUDU-2191 (12/n): Hive Metastore notification log event listener
This commit adds a notification log event listener to the Hive Metastore
integration, and shifts how the catalog manager handles table metadata
when the HMS integration is enabled. The leader master now listens for
drop table and alter table events in the HMS, and applies them to the
Kudu catalog as necessary. The latest handled notification log event
index is recorded in the sys-catalog. When the HMS integration is
enabled, the HMS is considered the source of truth for the table
namespace.
As a result, rename and delete RPCs are handled specially to ensure they
are applied to the HMS first, and only once they are committed in the
HMS are they applied to the Kudu catalog, through the new notification
log listener. Alter table operations which include a table rename and
other changes are handled by first performing the rename, then
performing the remaining changes. As a result, the alter table operation
as a whole is no longer applied atomically.
Testing: this is a hard component to test, because it is necessarily
tied to the catalog manager. I've added tests for specific scenarios in
master_hms-itest, however I'm aware that not all codepaths in the
listener are covered. master-stress-test and alter_table-randomized-test
are now parameterized to run with the HMS integration enabled, which has
been effective at revealing issues.
The HMS integration has known issues with capital letters in table
names. A follow-up commit will introduce a fix and tests for this issue.
Change-Id: I32ed099c44a593ffe514152135957018f21ed775
---
M src/kudu/hms/hms_catalog.cc
M src/kudu/hms/hms_catalog.h
M src/kudu/hms/hms_client-test.cc
M src/kudu/hms/mini_hms.cc
M src/kudu/integration-tests/CMakeLists.txt
M src/kudu/integration-tests/alter_table-randomized-test.cc
M src/kudu/integration-tests/master-stress-test.cc
M src/kudu/integration-tests/master_failover-itest.cc
M src/kudu/integration-tests/master_hms-itest.cc
M src/kudu/master/CMakeLists.txt
M src/kudu/master/catalog_manager.cc
M src/kudu/master/catalog_manager.h
A src/kudu/master/hms_notification_log_listener.cc
A src/kudu/master/hms_notification_log_listener.h
M src/kudu/master/master.proto
M src/kudu/master/sys_catalog.cc
M src/kudu/master/sys_catalog.h
M src/kudu/util/test_util.cc
18 files changed, 1,127 insertions(+), 225 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/13/8313/33
--
To view, visit http://gerrit.cloudera.org:8080/8313
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I32ed099c44a593ffe514152135957018f21ed775
Gerrit-Change-Number: 8313
Gerrit-PatchSet: 33
Gerrit-Owner: Dan Burkert <danburk...@apache.org>
Gerrit-Reviewer: Adar Dembo <a...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <danburk...@apache.org>
Gerrit-Reviewer: Hao Hao <hao....@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
