Hello Tidy Bot, Kudu Jenkins, Adar Dembo, Hao Hao,

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/8313

to look at the new patch set (#25).

Change subject: KUDU-2191 (12/n): Hive Metastore notification log event listener
......................................................................

KUDU-2191 (12/n): Hive Metastore notification log event listener

This commit adds a notification log event listener to the Hive Metastore
integration, and shifts how the catalog manager handles table metadata
when the HMS integration is enabled. The leader master now listens for
drop table and alter table events in the HMS, and applies them to the
Kudu catalog as necessary. The latest handled notification log event
index is recorded in the sys-catalog. When the HMS integration is
enabled, the HMS is considered the source of truth for the table
namespace. As a result, rename and delete RPCs are handled specially to
ensure they are applied to the HMS first, and only once they are
committed in the HMS are they applied to the Kudu catalog, through the
new notification log listener.

Testing: this is a hard component to test, because it is necessarily
tied to the catalog manager. I've added tests for specific scenarios in
master_hms-itest, however I'm aware that not all codepaths in the
listener are covered. master-stress-test and alter_table-randomized-test
are now parameterized to run with the HMS integration enabled, which has
been effective at revealing issues.

The HMS integration has known issues with capital letters in table
names. A follow-up commit will introduce a fix and tests for this
issue.

Change-Id: I32ed099c44a593ffe514152135957018f21ed775
---
M src/kudu/hms/hms_catalog.cc
M src/kudu/hms/hms_catalog.h
M src/kudu/hms/hms_client-test.cc
M src/kudu/hms/mini_hms.cc
M src/kudu/integration-tests/alter_table-randomized-test.cc
M src/kudu/integration-tests/master-stress-test.cc
M src/kudu/integration-tests/master_failover-itest.cc
M src/kudu/integration-tests/master_hms-itest.cc
M src/kudu/master/CMakeLists.txt
M src/kudu/master/catalog_manager.cc
M src/kudu/master/catalog_manager.h
A src/kudu/master/hms_notification_log_listener.cc
A src/kudu/master/hms_notification_log_listener.h
M src/kudu/master/master.proto
M src/kudu/master/sys_catalog.cc
M src/kudu/master/sys_catalog.h
M src/kudu/util/test_util.cc
17 files changed, 1,092 insertions(+), 219 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/13/8313/25
--
To view, visit http://gerrit.cloudera.org:8080/8313
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I32ed099c44a593ffe514152135957018f21ed775
Gerrit-Change-Number: 8313
Gerrit-PatchSet: 25
Gerrit-Owner: Dan Burkert <danburk...@apache.org>
Gerrit-Reviewer: Adar Dembo <a...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <danburk...@apache.org>
Gerrit-Reviewer: Hao Hao <hao....@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot

Reply via email to