Hello Tidy Bot, Kudu Jenkins, Adar Dembo, Hao Hao, I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/8313 to look at the new patch set (#18). Change subject: KUDU-2191 (12/n): Hive Metastore notification log event listener ...................................................................... KUDU-2191 (12/n): Hive Metastore notification log event listener This commit adds a notification log event listener to the Hive Metastore integration, and shifts how the catalog manager handles table metadata when the HMS integration is enabled. The leader master now listens for drop table and alter table events in the HMS, and applies them to the Kudu catalog as necessary. The latest handled notification log event index is recorded in the sys-catalog. When the HMS integration is enabled, the HMS is considered the source of truth for the table namespace. As a result, rename and delete RPCs are handled specially to ensure they are applied to the HMS first, and only once they are committed in the HMS are they applied to the Kudu catalog, through the new notification log listener. Testing: this is a hard component to test, because it is necessarily tied to the catalog manager. I've added tests for specific scenarios in master_hms-itest, however I'm aware that not all codepaths in the listener are covered. master-stress-test is now parameterized to run with the HMS integration enabled, since this was the 'missing piece' which allows the catalogs to remain (eventually) consistent in the presence of master crashes. The HMS integration has known issues with capital letters in table names. A follow-up commit will introduce a fix and tests for this issue. Change-Id: I32ed099c44a593ffe514152135957018f21ed775 --- M src/kudu/hms/hms_catalog.cc M src/kudu/hms/hms_catalog.h M src/kudu/integration-tests/master-stress-test.cc M src/kudu/integration-tests/master_failover-itest.cc M src/kudu/integration-tests/master_hms-itest.cc M src/kudu/master/CMakeLists.txt M src/kudu/master/catalog_manager.cc M src/kudu/master/catalog_manager.h A src/kudu/master/hms_notification_log_listener.cc A src/kudu/master/hms_notification_log_listener.h M src/kudu/master/master.proto M src/kudu/master/sys_catalog.cc M src/kudu/master/sys_catalog.h M src/kudu/util/test_util.cc 14 files changed, 1,037 insertions(+), 207 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/13/8313/18 -- To view, visit http://gerrit.cloudera.org:8080/8313 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I32ed099c44a593ffe514152135957018f21ed775 Gerrit-Change-Number: 8313 Gerrit-PatchSet: 18 Gerrit-Owner: Dan Burkert <danburk...@apache.org> Gerrit-Reviewer: Adar Dembo <a...@cloudera.com> Gerrit-Reviewer: Dan Burkert <danburk...@apache.org> Gerrit-Reviewer: Hao Hao <hao....@cloudera.com> Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tidy Bot