Vidit Gupta created HIVE-30052:
----------------------------------
Summary: CachedStore blocks all metastore RPCs during prewarm due
to shared class monitor
Key: HIVE-30052
URL: https://issues.apache.org/jira/browse/HIVE-30052
Project: Hive
Issue Type: Bug
Components: Standalone Metastore
Affects Versions: 4.2.0
Reporter: Vidit Gupta
Assignee: Vidit Gupta
*CachedStore.triggerPreWarm()* and *CachedStore.startCacheUpdateService()* are
both static synchronized, so they share the CachedStore class monitor. The
background update thread holds that monitor for the entire prewarm. Since
setConf() calls startCacheUpdateService() on every RawStore construction and
HMS constructs a thread-local RawStore per Thrift worker on first use every
incoming RPC on a cold-started HMS blocks until prewarm completes, even though
all read paths already fall back to the raw store correctly when the cache is
not yet warm.
On our deployment (21k whitelisted tables, Hive 4.2.0), HMS accepted
connections but served zero RPCs for the full ~10-minute prewarm. Thread dumps
show Thrift worker threads BLOCKED on the class monitor entering
startCacheUpdateService. The affected code is unchanged on master.
Fix: move startCacheUpdateService/stopCacheUpdateService onto a dedicated lock
object (they only guard executor init/shutdown and share no state with
prewarm), and make cacheUpdateMaster volatile with a lock-free fast path. After
the fix, RPCs are served from the backing DB during prewarm (get_table ~30ms in
our tests) and the cache takes over when prewarm completes. A deterministic
regression test is included. PR to follow.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
