Sai Hemanth Gantasala has uploaded this change for review. ( http://gerrit.cloudera.org:8080/20367
Change subject: IMPALA-10976: Sync db/table in catalogD to latest HMS event id for all DDLS from impala shell ...................................................................... IMPALA-10976: Sync db/table in catalogD to latest HMS event id for all DDLS from impala shell The idea is that when any DDL operation is performed from Impala shell, it also syncs the db/table to its latest event ID as per HMS. This way updates to a db/table's are applied in the same order as they appear in the Notification log in HMS which ensures consistency. Currently catalogD applies any updates received from Impala shell in place. Instead it should perform an HMS operation first and then replay all the HMS events since the last synced event id. Implementation: when the enable_sync_to_latest_event_on_ddls flag is set to true, we fetch the latest snapshot of db/table and update it in the catalogD's cache. Once HIVE-27499 is implemented, we can update the snapshot only if there are any pending events. Currently, there is no efficient way to identify if there are pending events for a db/table. Set enable_sync_to_latest_event_on_ddls to true to use this feature. Testing: Added few tests in the MetaStoreEventProcessorForTest to verify this feature. Change-Id: Ia250d0a943838086c187e5cb7c60035e5a564bbf --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 4 files changed, 151 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/67/20367/1 -- To view, visit http://gerrit.cloudera.org:8080/20367 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ia250d0a943838086c187e5cb7c60035e5a564bbf Gerrit-Change-Number: 20367 Gerrit-PatchSet: 1 Gerrit-Owner: Sai Hemanth Gantasala <[email protected]>
