Hello Quanlong Huang, Zoltan Borok-Nagy, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/19484
to look at the new patch set (#7).
Change subject: IMPALA-11822: Optimize the Refresh/Invalidate event processing
by skipping unnecessary events
......................................................................
IMPALA-11822: Optimize the Refresh/Invalidate event processing by skipping
unnecessary events
Added a new variable 'lastRefreshEventId' in the
catalogD's table/partition object to store the
latest event id before loading the table/partition.
This will be updated frequently based on refresh
or invalidate commands. This variable can be used
in the event processor to decide whether to process
or skip the reload event by comparing it with the
current event id. It is enough to store the refresh
event's event id, invalidate event anyway flushes
out the object from cache.
Note: Need to enable two configs for this
optimization to work:
1) enable_reload_events=true
2) enable_sync_to_latest_event_on_ddls=true
Testing: Added a test to fire few reload events via
HMS API and then verify in the event processor that
some older events are skipped.
Change-Id: I905957683a96c3ea01ab4bf043d6658ce37b7574
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/catalog/TableLoader.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M tests/custom_cluster/test_events_custom_configs.py
8 files changed, 135 insertions(+), 13 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/84/19484/7
--
To view, visit http://gerrit.cloudera.org:8080/19484
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I905957683a96c3ea01ab4bf043d6658ce37b7574
Gerrit-Change-Number: 19484
Gerrit-PatchSet: 7
Gerrit-Owner: Sai Hemanth Gantasala <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Quanlong Huang <[email protected]>
Gerrit-Reviewer: Sai Hemanth Gantasala <[email protected]>
Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
