Yida Wu has uploaded this change for review. ( http://gerrit.cloudera.org:8080/23942
Change subject: IMPALA-14230: Add catch-up mode for event processing ...................................................................... IMPALA-14230: Add catch-up mode for event processing When HMS events fall far behind (e.g. during HA failover), event handling can become a serious bottleneck especially in the single threaded legacy mode, heavy operations such as table reload could take minutes that blocks the whole event processing. This patch introduces a configurable catch-up mode. If event lag exceeds the new added flag hms_event_catchup_threshold_s (default 1800s), the event processor switches to a faster way by invalidating the table instead of performing every event to avoid unnecessary heavy table reload to speed up the event processing. Catch-up mode is applied only to table events that may trigger heavy operations. Lightweight events such as create/drop table and database events are excluded since they are already fast. Testing: Added test_catchup_mode_triggered_by_delay. Passed exhaustive tests. Change-Id: Ib906c06346d5d3159999eeac632e1318bc060065 --- M be/src/catalog/catalog-server.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M tests/custom_cluster/test_events_custom_configs.py 6 files changed, 82 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/42/23942/1 -- To view, visit http://gerrit.cloudera.org:8080/23942 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ib906c06346d5d3159999eeac632e1318bc060065 Gerrit-Change-Number: 23942 Gerrit-PatchSet: 1 Gerrit-Owner: Yida Wu <[email protected]>
