[
https://issues.apache.org/jira/browse/IMPALA-12152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17929405#comment-17929405
]
ASF subversion and git services commented on IMPALA-12152:
----------------------------------------------------------
Commit b410a90b4707be2e08dcb6c02b9473aa9ade8d96 in impala's branch
refs/heads/master from stiga-huang
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=b410a90b4 ]
IMPALA-12152: Add query options to wait for HMS events sync up
It's a common scenario to run Impala queries after the dependent
external changes are done. E.g. running COMPUTE STATS on a table after
Hive/Spark jobs ingest some new data to it. Currently, it's unsafe to
run the Impala queries immediately after the Hive/Spark jobs finish
since EventProcessor might have a long lag in applying the HMS events.
Note that running REFRESH/INVALIDATE on the table can also solve the
problem. But one of the motivation of EventProcessor is to get rid of
such Impala specific commands.
This patch adds a mechanism to let query planning wait until the
metadata is synced up. Two new query options are added:
- SYNC_HMS_EVENTS_WAIT_TIME_S configures the timeout in seconds for
waiting. It's 0 by default, which disables the waiting mechanism.
- SYNC_HMS_EVENTS_STRICT_MODE controls the behavior if we can't wait
for metadata to be synced up, e.g. when the waiting times out or
EventProcessor is in ERROR state. It defaults to false (non-strict
mode). In the strict mode, coordinator will fail the query. In the
non-strict mode, coordinator will start planning with a warning
message in profile (and in client outputs if the client consumes the
get_log results, e.g. in impala-shell).
Example usage - query the table after inserting into dynamic partitions
in Hive. We don't know what partitions are modified so running REFRESH
in Impala is inefficient since it reloads all partitions.
hive> insert into tbl partition(p) select * from tbl2;
impala> set sync_hms_events_wait_time_s=300;
impala> select * from tbl;
With this new feature, let catalogd reload the updated partitions based
on HMS events, which is more efficient than REFRESH. The wait time can
be set to the largest lag of event processing that has been observed in
the cluster. Note the lag of event processing is shown as the "Lag time"
in the /events page of catalogd WebUI and "events-processor.lag-time" in
the /metrics page. Users can monitor it to get a sense of the lag.
Some timeline items are added in query profile for this waiting
mechanism, e.g.
A succeeded wait:
Query Compilation: 937.279ms
- Synced events from Metastore: 909.162ms (909.162ms)
- Metadata of all 1 tables cached: 911.005ms (1.843ms)
- Analysis finished: 919.600ms (8.595ms)
A failed wait:
Query Compilation: 1s321ms
- Continuing without syncing Metastore events: 40.883ms (40.883ms)
- Metadata load started: 41.618ms (735.633us)
Added a histogram metric, impala-server.wait-for-hms-event-durations-ms,
to track the duration of this waiting.
--------
Implementation
A new catalogd RPC, WaitForHmsEvent, is added to CatalogService API so
that coordinator can wait until catalogd processes the latest event when
this RPC is triggered. Query planning starts or fails after this RPC
returns. The RPC request contains the potential dbs/tables that are
required by the query. Catalogd records the latest event id when it
receives this RPC. When the last synced event id reaches this, catalogd
returns the catalog updates to the coordinator in the RPC response.
Before that, the RPC thread is in a waiting loop that sleeps in a
configurable interval. It's configured by a hidden flag,
hms_event_sync_sleep_interval_ms (defaults to 100).
Entry-point functions
- Frontend#waitForHmsEvents()
- CatalogServiceCatalog#waitForHmsEvent()
Some statements don't need to wait for HMS events, e.g. CREATE/DROP ROLE
statements. This patch adds an overrided method, requiresHmsMetadata(),
in each Statement to mark whether they can skip HMS event sync.
Test side changes:
- Some test codes use EventProcessorUtils.wait_for_event_processing()
to wait for HMS events being synced up before running a query. Now
they are updated to just use these new query options in the query.
- Note that we still need wait_for_event_processing() in test codes
that verify metrics after HMS events are synced up.
--------
Limitation
Currently, UPDATE_TBL_COL_STAT_EVENT, UPDATE_PART_COL_STAT_EVENT,
OPEN_TXN events are ignored by the event processor. If the latest event
happens to be in these types and there are no more other events, the
last synced event id can never reach the latest event id. We need to fix
last synced event id to also consider ignored events (IMPALA-13623).
The current implementation waits for the event id when the
WaitForHmsEvent RPC is received at catalogd side. We can improve it
by leveraging HIVE-27499 to efficiently detect whether the given
dbs/tables have unsynced events and just wait for the *largest* id
of them. Dbs/tables without unsynced events don't need to block query
planning. However, this only works for non-transactional tables.
Transactional tables might be modified by COMMIT_TXN or ABORT_TXN events
which don't have the table names. So even with HIVE-27499, we can't
determine whether a transactional table has pending events. IMPALA-13684
will target on improving this on non-transactional tables.
Tests
- Add test to verify planning waits until catalogd is synced with HMS
changes.
- Add test on the error handling when HMS event processing is disabled
Change-Id: I36ac941bb2c2217b09fcfa2eb567b011b38efa2a
Reviewed-on: http://gerrit.cloudera.org:8080/20131
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Add query option to wait for events sync up
> -------------------------------------------
>
> Key: IMPALA-12152
> URL: https://issues.apache.org/jira/browse/IMPALA-12152
> Project: IMPALA
> Issue Type: New Feature
> Components: Catalog
> Reporter: Quanlong Huang
> Assignee: Quanlong Huang
> Priority: Critical
> Labels: catalog-2024
>
> Event-processor is designed to get rid of manual RT/IM (RefreshTable /
> InvalidateMetadata) commands that sync up with external HMS modifications.
> However, event processing could be delayed. Queries might still see stale
> metadata if the event-processor is lagging behind. We should provide a
> mechanism to let query planning wait until the metadata is synced up. Users
> can turn it on for sensitive queries that depend on external modifications.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
