Vidit Gupta created HIVE-30053:
----------------------------------
Summary: Support multi-threaded CachedStore prewarm
Key: HIVE-30053
URL: https://issues.apache.org/jira/browse/HIVE-30053
Project: Hive
Issue Type: Bug
Components: Standalone Metastore
Affects Versions: 4.2.0
Reporter: Vidit Gupta
Assignee: Vidit Gupta
CachedStore prewarm is single-threaded: one background thread sequentially
fetches every whitelisted table with its partitions, statistics and
constraints. On large warehouses this makes the warmup window very long (~10
minutes for 21k tables on our deployment), during which the cache serves
nothing (with HIVE-30052 fixed, requests fall through to the raw store, but at
DB latency and DB load).
This issue adds a config-gated parallel prewarm:
metastore.cached.rawstore.prewarm.threads (default 1 = current single-threaded
behavior, zero change unless opted in). With N > 1, N worker threads — each
with its own RawStore instance, mirroring how CacheUpdateMasterWork creates its
private store — drain the existing TablesPendingPrewarm stack, preserving the
prioritizeTableForPrewarm hot-table promotion and the memory-full early stop
(shared flag, completePrewarm called exactly once). Workers exist only for the
duration of prewarm. Each worker uses one DB connection from the shared pool,
so the value should be kept below the connection pool size
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
