[
https://issues.apache.org/jira/browse/IMPALA-14074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18004906#comment-18004906
]
ASF subversion and git services commented on IMPALA-14074:
----------------------------------------------------------
Commit da190f1d867e359f6cfb8cf80dcaa3d278ca9c7e in impala's branch
refs/heads/master from stiga-huang
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=da190f1d8 ]
IMPALA-14074: Warmup metadata cache in catalogd for critical tables
*Background*
Catalogd starts with a cold metadata cache - only the db/table names and
functions are loaded. Metadata of a table is unloaded until there are
queries submitted on the table. The first query will suffer from the
delay of loading metadata. There is a flag,
--load_catalog_in_background, to let catalogd eagerly load metadata of
all tables even if no queries come. Catalogd may load metadata for
tables that are possibly never used, potentially increasing catalog size
and consequently memory usage. So this flag is turned off by default and
not recommended to be used in production.
Users do need the metadata of some critical tables to be loaded. Before
that the service is considered not ready since important queries might
fail in timeout. When Catalogd HA is enabled, it’s also required that
the standby catalogd has an up-to-date metadata cache to smoothly take
over the active one when failover happens.
*New Flags*
This patch adds a startup flag for catalogd to specify a config file
containing tables that users want their metadata to be loaded. Catalogd
adds them to the table loading queue in background when a catalog reset
happens, i.e. at catalogd startup or global INVALIDATE METADATA runs.
The flag is --warmup_tables_config_file. The value can be a path in the
local FS or in remote storage (e.g. HDFS). E.g.
--warmup_tables_config_file=file:///opt/impala/warmup_table_list.txt
--warmup_tables_config_file=hdfs:///tmp/warmup_table_list.txt
Each line in the config file can be a fully qualified table name or a
wildcard under a db, e.g. "tpch.*". Catalogd loads the table names at
startup and schedules loading on them after a reset of the catalog. The
scheduling order is based on the order in the config file. So important
tables can be put first. Comments start with "#" or "//" are ignored in
the config file.
Another flag, --keeps_warmup_tables_loaded (defaults to false), is added
to control whether to reload the table after it’s been invalidated,
either by an explicit INVALIDATE METADATA <table> command or implicitly
invalidated by CatalogdTableInvalidator or HMS RELOAD events.
When CatalogdTableInvalidator is enabled with
--invalidate_tables_on_memory_pressure=true, users shouldn’t set
keeps_warmup_tables_loaded to true if the catalogd heap size is not
enough to cache metadata of all these tables. Otherwise, these tables
will keep being loaded and invalidated.
*Catalogd HA Changes*
When Catalogd HA is enabled, the standby catalogd will also reset its
catalog and start loading metadata of these tables, after the HA state
(active/standby) is determined. Standby catalogd keeps its metadata
cache up-to-date by applying HMS notification events. To support a
warmed up switch, --catalogd_ha_reset_metadata_on_failover should be set
to false.
*Limitation*
The standby catalogd could still have a stale cache if there are
operations in the active catalogd that don’t trigger HMS notification
events, or if the HMS notification event is not applied correctly. E.g.
Adding a new native function generates an ALTER_DATABASE event, but when
applying the event, native function list of the db is not refreshed
(IMPALA-14210). These will be resolved in separate JIRAs.
*Test*
- Added FE unit tests.
- Added e2e test for local/hdfs config files.
- Added e2e test to verify the standby catalogd has a warmed up cache
when failover happens.
Change-Id: I2d09eae1f12a8acd2de945984d956d11eeee1ab6
Reviewed-on: http://gerrit.cloudera.org:8080/23155
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Standby catalogd should warmup the metadata cache
> -------------------------------------------------
>
> Key: IMPALA-14074
> URL: https://issues.apache.org/jira/browse/IMPALA-14074
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Reporter: Quanlong Huang
> Assignee: Quanlong Huang
> Priority: Critical
>
> Users might want to warm up the metadata cache for some important tables
> after starting catalogd, which could take hours in a large warehouse. When
> Catalogd HA is enabled and a failover happens, the standby catalogd becomes
> active with a cold cache. This requires users to warm up the cache again.
> During this period, queries suffer from poor performance and might fail in
> timeout.
> We can provide a configuration file loaded in catalogd startup that contains
> one table at a line. E.g. --preload_metadata_table_list_file=table_list.txt.
> Catalogd triggers metadata loading of those tables in the background. Then
> users don't need to explictly run some queries to warm up the cache. Since
> the tables are loaded, HMS notification events on them won't be skipped in
> the standby catalogd. In this way, the standby catalogd can keep the cache
> up-to-date.
> These tables will be added into the table loading queue, just like how
> PrioritizeLoadRequest triggered by queries does. So the concurrency is still
> controlled by num_metadata_loading_threads.
> Catalogd should exponse metrics to indicate whether the loading of these
> tables is done. E.g. num-preload-metadata-tasks for all valid table names in
> the list, and num-preload-metadata-tasks-done for loaded tables. When these
> two metrics are equal, the warmup is done.
> Metadata warmup should also happens after global INVALIDATE METADATA, similar
> to startup.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
