[
https://issues.apache.org/jira/browse/IMPALA-14074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18007938#comment-18007938
]
ASF subversion and git services commented on IMPALA-14074:
----------------------------------------------------------
Commit 64abca481ffefcc67cee9e8c20de51e68238be95 in impala's branch
refs/heads/master from stiga-huang
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=64abca481 ]
IMPALA-14227: In HA failover, passive catalogd should apply pending HMS events
before being active
After IMPALA-14074, the passive catalogd can have a warmed up metadata
cache during failover (with catalogd_ha_reset_metadata_on_failover=false
and a non-empty warmup_tables_config_file). However, it could still use
a stale metadata cache when some pending HMS events generated by the
previous active catalogd are not applied yet.
This patch adds a wait during HA failover to ensure HMS events before
the failover happens are all applied on the new active catalogd. The
timeout is configured by a new flag which defaults to 300 (5 minutes):
catalogd_ha_failover_catchup_timeout_s. When timeout happens, by default
catalogd will fallback to resetting all metadata. Users can decide
whether to reset or continue using the current cache. This is configured
by another flag, catalogd_ha_reset_metadata_on_failover_catchup_timeout.
Since the passive catalogd depends on HMS event processing to keep its
metadata up-to-date with the active catalogd, this patch adds validation
to avoid starting catalogd with catalogd_ha_reset_metadata_on_failover
set to false and hms_event_polling_interval_s <= 0.
This patch also makes catalogd_ha_reset_metadata_on_failover a
non-hidden flag so it's shown in the /varz web page.
Tests:
- Ran test_warmed_up_metadata_after_failover 200 times. Without the
fix, it usually fails in several runs.
- Added new tests for the new flags.
Change-Id: Icf4fcb0e27c14197f79625749949b47c033a5f31
Reviewed-on: http://gerrit.cloudera.org:8080/23174
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Standby catalogd should warmup the metadata cache
> -------------------------------------------------
>
> Key: IMPALA-14074
> URL: https://issues.apache.org/jira/browse/IMPALA-14074
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Reporter: Quanlong Huang
> Assignee: Quanlong Huang
> Priority: Critical
> Fix For: Impala 5.0.0
>
>
> Users might want to warm up the metadata cache for some important tables
> after starting catalogd, which could take hours in a large warehouse. When
> Catalogd HA is enabled and a failover happens, the standby catalogd becomes
> active with a cold cache. This requires users to warm up the cache again.
> During this period, queries suffer from poor performance and might fail in
> timeout.
> We can provide a configuration file loaded in catalogd startup that contains
> one table at a line. E.g. --preload_metadata_table_list_file=table_list.txt.
> Catalogd triggers metadata loading of those tables in the background. Then
> users don't need to explictly run some queries to warm up the cache. Since
> the tables are loaded, HMS notification events on them won't be skipped in
> the standby catalogd. In this way, the standby catalogd can keep the cache
> up-to-date.
> These tables will be added into the table loading queue, just like how
> PrioritizeLoadRequest triggered by queries does. So the concurrency is still
> controlled by num_metadata_loading_threads.
> Catalogd should exponse metrics to indicate whether the loading of these
> tables is done. E.g. num-preload-metadata-tasks for all valid table names in
> the list, and num-preload-metadata-tasks-done for loaded tables. When these
> two metrics are equal, the warmup is done.
> Metadata warmup should also happens after global INVALIDATE METADATA, similar
> to startup.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
