[
https://issues.apache.org/jira/browse/HIVE-30053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhihua Deng resolved HIVE-30053.
--------------------------------
Fix Version/s: 4.3.0
Resolution: Fixed
> Support multi-threaded CachedStore prewarm
> -------------------------------------------
>
> Key: HIVE-30053
> URL: https://issues.apache.org/jira/browse/HIVE-30053
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
> Affects Versions: 4.2.0
> Reporter: Vidit Gupta
> Assignee: Vidit Gupta
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.3.0
>
>
> CachedStore prewarm is single-threaded: one background thread sequentially
> fetches every whitelisted table with its partitions, statistics and
> constraints. On large warehouses this makes the warmup window very long (~10
> minutes for 21k tables on our deployment), during which the cache serves
> nothing (with HIVE-30052 fixed, requests fall through to the raw store, but
> at DB latency and DB load).
> This issue adds a config-gated parallel prewarm:
> metastore.cached.rawstore.prewarm.threads (default 1 = current
> single-threaded behavior, zero change unless opted in). With N > 1, N worker
> threads — each with its own RawStore instance, mirroring how
> CacheUpdateMasterWork creates its private store — drain the existing
> TablesPendingPrewarm stack, preserving the prioritizeTableForPrewarm
> hot-table promotion and the memory-full early stop (shared flag,
> completePrewarm called exactly once). Workers exist only for the duration of
> prewarm. Each worker uses one DB connection from the shared pool, so the
> value should be kept below the connection pool size
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
