Hello [email protected], Sai Hemanth Gantasala, Michael Smith, Csaba
Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20131
to look at the new patch set (#19).
Change subject: IMPALA-12152: Add query option to wait for events sync up
......................................................................
IMPALA-12152: Add query option to wait for events sync up
Event-processor is designed to get rid of manual RT/IM (RefreshTable /
InvalidateMetadata) commands that sync up with external HMS
modifications. However, event processing could be delayed. Queries might
still see stale metadata if the event-processor is lagging behind.
This patch adds a mechanism to let query planning wait until the
metadata is synced up. To be specific, coordinator will not start
planning until the last synced event id of catalogd reaches the latest
event id when the query is submitted. A new catalogd RPC,
WaitForHmsEvent, is added for this. Coordinator parses the query and
sends potential dbs/tables that are required by the query. Catalogd
records the latest event id and returns the required catalog updates
once it catches up with that event id. Coordinator then applies the
catalog updates and continues query planning.
Note that the current implementation waits for the latest event id when
the WaitForHmsEvent RPC is received at catalogd side. We can improve it
once HIVE-27499 is resolved, so we can efficiently detect whether some
given dbs/tables have unsynced events and just wait for the *largest* id
of them. Dbs/tables without unsynced events don't need to block query
planning.
A new query option, sync_hms_events_wait_time_s, is added to configure
the timeout for waiting. It's 0 by default, which disables the waiting
mechanism. Users can turn it on for sensitive queries that depend on
external modifications.
Another new query option, sync_hms_events_strict_mode, is added to
control the behavior on errors, e.g. timeout or event-processor in error
state. It defaults to false (non-strict mode). In the strict mode,
coordinator will fail the query if it fails to wait for HMS events to be
synced in catalogd. In the non-strict mode, coordinator will start
planning with a warning message in profile (and in client outputs if the
client consumes the get_log results, e.g. in impala-shell).
Example usage - query the table after inserting into dynamic partitions
in Hive. We don't know what partitions are modified so running REFRESH
in Impala is inefficient since it reloads all partitions.
hive> insert into tbl partition(p) select * from tbl2;
impala> set sync_hms_events_wait_time_s=300;
impala> select * from tbl;
With this new feature, let catalogd reload the updated partitions based
on HMS events, which is more efficient than REFRESH. The wait time can
be set to the largest lag of event processing that has been observed in
the cluster.
Some timeline items are added in query profile for this waiting, e.g.
A succeeded wait:
Query Compilation: 937.279ms
- Synced events from Metastore: 909.162ms (909.162ms)
- Metadata of all 1 tables cached: 911.005ms (1.843ms)
- Analysis finished: 919.600ms (8.595ms)
A failed wait:
Query Compilation: 1s321ms
- Failed to sync events from Metastore: 40.883ms (40.883ms)
- Metadata load started: 41.618ms (735.633us)
For better debuggability in tests, add logs in run_stmt_in_hive to print
the Hive statements.
Limitation:
Strict mode might fail in timeout if the latest event is skipped by the
event processor, thus the last synced event id won't reach the latest
event it until there are new events generated in HMS. Currently,
OPEN_TXN events are skipped by event processor.
Tests
- Add test to verify planning waits until catalogd is synced with HMS
changes.
- Add test on the error handling when HMS event processing is disabled
- There are some existing tests that use
EventProcessorUtils.wait_for_event_processing() to wait until events
synced. Modify them to use the new query option in queries need this.
Change-Id: I36ac941bb2c2217b09fcfa2eb567b011b38efa2a
---
M be/src/catalog/catalog-server.cc
M be/src/catalog/catalog-service-client-wrapper.h
M be/src/catalog/catalog.cc
M be/src/catalog/catalog.h
M be/src/exec/catalog-op-executor.cc
M be/src/exec/catalog-op-executor.h
M be/src/runtime/coordinator.cc
M be/src/service/fe-support.cc
M be/src/service/impala-server.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M common/thrift/CatalogService.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/analysis/AdminFnStmt.java
M fe/src/main/java/org/apache/impala/analysis/AlterDbStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateDbStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateDropRoleStmt.java
M fe/src/main/java/org/apache/impala/analysis/DescribeDbStmt.java
M fe/src/main/java/org/apache/impala/analysis/DropDbStmt.java
M fe/src/main/java/org/apache/impala/analysis/GrantRevokeRoleStmt.java
M fe/src/main/java/org/apache/impala/analysis/SetStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowDataSrcsStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowFunctionsStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowGrantPrincipalStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowRolesStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowTablesOrViewsStmt.java
M fe/src/main/java/org/apache/impala/analysis/StatementBase.java
M fe/src/main/java/org/apache/impala/catalog/Catalog.java
M fe/src/main/java/org/apache/impala/catalog/CatalogDeltaLog.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java
M fe/src/main/java/org/apache/impala/catalog/events/ExternalEventsProcessor.java
M
fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
M fe/src/main/java/org/apache/impala/catalog/events/NoOpEventProcessor.java
M fe/src/main/java/org/apache/impala/service/FeSupport.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/JniCatalog.java
M tests/common/impala_test_suite.py
M tests/custom_cluster/test_events_custom_configs.py
M tests/custom_cluster/test_hive_parquet_codec_interop.py
M tests/custom_cluster/test_kudu.py
M tests/metadata/test_event_processing.py
M tests/metadata/test_event_processing_base.py
M tests/metadata/test_metadata_query_statements.py
M tests/metadata/test_recover_partitions.py
48 files changed, 875 insertions(+), 110 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/20131/19
--
To view, visit http://gerrit.cloudera.org:8080/20131
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I36ac941bb2c2217b09fcfa2eb567b011b38efa2a
Gerrit-Change-Number: 20131
Gerrit-PatchSet: 19
Gerrit-Owner: Quanlong Huang <[email protected]>
Gerrit-Reviewer: Anonymous Coward <[email protected]>
Gerrit-Reviewer: Csaba Ringhofer <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Michael Smith <[email protected]>
Gerrit-Reviewer: Quanlong Huang <[email protected]>
Gerrit-Reviewer: Sai Hemanth Gantasala <[email protected]>
