Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/19484 )
Change subject: IMPALA-11822: Optimize the Refresh/Invalidate event processing by skipping unnecessary events ...................................................................... IMPALA-11822: Optimize the Refresh/Invalidate event processing by skipping unnecessary events Added a new variable 'lastRefreshEventId' in the catalogD's table/partition object to store the latest event id before loading the table/partition. This will be updated frequently based on refresh or invalidate commands. This variable can be used in the event processor to decide whether to process or skip the reload event by comparing it with the current event id. It is enough to store the refresh event's event id, invalidate event anyway flushes out the object from cache. Note: Need to enable two configs for this optimization to work: 1) enable_reload_events=true 2) enable_sync_to_latest_event_on_ddls=true Testing: Added a test to fire few reload events via HMS API and then verify in the event processor that some older events are skipped. Change-Id: I905957683a96c3ea01ab4bf043d6658ce37b7574 Reviewed-on: http://gerrit.cloudera.org:8080/19484 Reviewed-by: Impala Public Jenkins <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/TableLoader.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M tests/custom_cluster/test_events_custom_configs.py 8 files changed, 135 insertions(+), 13 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/19484 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I905957683a96c3ea01ab4bf043d6658ce37b7574 Gerrit-Change-Number: 19484 Gerrit-PatchSet: 10 Gerrit-Owner: Sai Hemanth Gantasala <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Reviewer: Sai Hemanth Gantasala <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
