Hello Quanlong Huang, Zoltan Borok-Nagy, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20022
to look at the new patch set (#10).
Change subject: IMPALA-11535: Skip older events in the event processor based on
the latestRefreshEventID
......................................................................
IMPALA-11535: Skip older events in the event processor based on the
latestRefreshEventID
Summary: If the table has been manually refreshed, all its events
happen before the manual REFRESH can be skipped. This happens when
catalogd is lagging behind in processing events. When processing an
event, we can check whether there are manual REFRESH executed after
its eventTime. In such case, we don't need to process the event to
refresh anything. This helps catalogd to catch up HMS events quickly.
Implementation details: Updated the lastRefreshEventId on the table or
partition whenever there is table or partition level refresh/load.
By comparing the lastRefreshEventId to current event id in the event
processor the older events can be skipped.
set enable_skipping_older_events to true to enable this optimization
Testing:
- Unit end-to-end test and unit test to test the functionality.
Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
---
M be/src/catalog/catalog-server.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M
fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M
fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
M tests/custom_cluster/test_events_custom_configs.py
11 files changed, 201 insertions(+), 9 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/22/20022/10
--
To view, visit http://gerrit.cloudera.org:8080/20022
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
Gerrit-Change-Number: 20022
Gerrit-PatchSet: 10
Gerrit-Owner: Sai Hemanth Gantasala <[email protected]>
Gerrit-Reviewer: Csaba Ringhofer <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Quanlong Huang <[email protected]>
Gerrit-Reviewer: Sai Hemanth Gantasala <[email protected]>
Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
