[
https://issues.apache.org/jira/browse/IMPALA-14230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18062741#comment-18062741
]
ASF subversion and git services commented on IMPALA-14230:
----------------------------------------------------------
Commit a99ccdacea7860af4dececc7d1f7560a15ccaf9e in impala's branch
refs/heads/master from Yida Wu
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=a99ccdace ]
IMPALA-14230: Add catch-up mode for event processing
When HMS events fall far behind (e.g. during HA failover), event
handling can become a serious bottleneck especially in the single
threaded legacy mode, heavy operations such as table reload could
take minutes that blocks the whole event processing.
This patch introduces a configurable catch-up mode. If event lag
exceeds the new added flag hms_event_catchup_threshold_s
(default 1800s), the event processor switches to a faster way by
invalidating the table instead of performing every event to avoid
unnecessary heavy table reload to speed up the event processing.
Catch-up mode is applied only to table events that may trigger heavy
operations, like reloading tables or operations that may involve
severe and slow lock contention. Lightweight events such as
create/drop table and database events are excluded since they are
already fast.
Testing:
Added test_catchup_mode_* tests in TestEventProcessingCatchupMode.
Passed exhaustive tests.
Change-Id: Ib906c06346d5d3159999eeac632e1318bc060065
Reviewed-on: http://gerrit.cloudera.org:8080/23942
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Add a "catch-up" mode to EventProcessor
> ---------------------------------------
>
> Key: IMPALA-14230
> URL: https://issues.apache.org/jira/browse/IMPALA-14230
> Project: IMPALA
> Issue Type: New Feature
> Components: Catalog
> Reporter: Quanlong Huang
> Assignee: Yida Wu
> Priority: Critical
>
> When HMS events processing has a large lag, we need EventProcessor to quickly
> catch up with HMS by simply processing the events as invalidations, i.e.
> falling back to the behavior before IMPALA-9101 that table level events just
> invalidate the table.
> This will also be useful in catalogd HA failover that the passive catalogd
> wants to quickly catch up with HMS.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
