Quanlong Huang has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/20131 )
Change subject: WIP IMPALA-12152: Add query option to wait for events sync up ...................................................................... WIP IMPALA-12152: Add query option to wait for events sync up Event-processor is designed to get rid of manual RT/IM (RefreshTable / InvalidateMetadata) commands that sync up with external HMS modifications. However, event processing could be delayed. Queries might still see stale metadata if the event-processor is lagging behind. This patch adds a mechanism to let query planning wait until the metadata is synced up. To be specific, coordinator will not start planning until the last synced event id of catalogd reaches the latest event id when the query is submitted. A new query option, sync_hms_events_wait_time_s, is added to configure the timeout for waiting. It's 0 by default, which disable the waiting mechanism. Users can turn it on for sensitive queries that depend on external modifications. Tests - Add test to verify planning waits until catalogd synced. - Add test on the error handling when HMS event processing is disabled Change-Id: I36ac941bb2c2217b09fcfa2eb567b011b38efa2a --- M be/src/catalog/catalog-server.cc M be/src/catalog/catalog-service-client-wrapper.h M be/src/catalog/catalog.cc M be/src/catalog/catalog.h M be/src/exec/catalog-op-executor.cc M be/src/exec/catalog-op-executor.h M be/src/runtime/coordinator.cc M be/src/runtime/coordinator.h M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/impala-server.cc M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/CatalogService.thrift M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/events/ExternalEventsProcessor.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/catalog/events/NoOpEventProcessor.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java M tests/common/impala_test_suite.py M tests/custom_cluster/test_events_custom_configs.py 23 files changed, 255 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/20131/3 -- To view, visit http://gerrit.cloudera.org:8080/20131 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I36ac941bb2c2217b09fcfa2eb567b011b38efa2a Gerrit-Change-Number: 20131 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang <[email protected]>
