Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/20648 )
Change subject: IMPALA-10987: Changing impala.disableHmsSync in Hive should not break event processing ...................................................................... IMPALA-10987: Changing impala.disableHmsSync in Hive should not break event processing Currently we require a global invalidate to reset the events processor if the events sync is re-enabled on a table from HMS. This patch eliminates the need to reset the catalog cache when events sync is re-enabled. Implementation details: when events sync is re-enabled on table via HMS 1) If the table exists in Impala, a) We can just invalidate the table, if the current event is greater than the create event id of the table, so that it is reloaded the first time query accesses it. b) Otherwise we can just ignore the event. 2) If the table doesn't exist in Impala, create a Incomplete table, if there is no entry in the event delete log for this table. Note: If the eventSync is disabled on a table, for all subsequent table events, ideally we should mark the table as stale if the table object is loaded, so that it is reloaded the next time query accesses it. But, since this approach has performance impact, the events will be ignored. Testing: 1) manually verified few scenarios. 2) Added test case for the above scenarios. Change-Id: I37055990be49e91462ebc98aa97009ca768a0072 Reviewed-on: http://gerrit.cloudera.org:8080/20648 Reviewed-by: Impala Public Jenkins <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java M tests/custom_cluster/test_events_custom_configs.py 3 files changed, 162 insertions(+), 59 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/20648 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I37055990be49e91462ebc98aa97009ca768a0072 Gerrit-Change-Number: 20648 Gerrit-PatchSet: 12 Gerrit-Owner: Sai Hemanth Gantasala <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Reviewer: Sai Hemanth Gantasala <[email protected]>
