Hi all,

While running the Hive 4.2.0 standalone metastore with CachedStore enabled
(~21k whitelisted tables), we found that a cold-started HMS accepts
connections but serves zero RPCs until prewarm completes (~10 minutes for
us).

Root cause: triggerPreWarm() and startCacheUpdateService() are both static
synchronized, so they share the CachedStore class monitor. The background
thread holds it for the entire prewarm, while every new Thrift worker
thread calls startCacheUpdateService via setConf when constructing its
thread-local RawStore — so all workers block before ever reaching the
existing serve-from-rawstore fallback paths. Thread dumps show the workers
BLOCKED on the class monitor. The code is unchanged on master.

Fix + regression test: https://github.com/apache/hive/pull/6783 (JIRA:
https://issues.apache.org/jira/browse/HIVE-30052). Verified on our
production deployment — after the fix, RPCs are served from the backing DB
in tens of milliseconds throughout prewarm, and the cache takes over once
prewarm completes.

We also prototyped a config-gated multi-threaded prewarm (default 1 =
current behavior; each worker uses its own RawStore instance): prewarm went
from 602s to 195s with 8 threads on the same deployment. If there's
interest I'll file a follow-up JIRA with that patch.

A review of PR #6783 would be much appreciated.

Thanks,
Vidit Gupta
Data Platform, Meesho

-- 
***
This communication is confidential, may be privileged, and is meant 
only for the intended recipient and purpose. No part of this email or any 
files transmitted with it can be shared, copied, forwarded, published 
online or offline, or used in any unauthorised manner. If you are not the 
intended recipient, please preserve the confidentiality of the contents, 
delete the e-mail and attachments (if any) from your system, and inform the 
sender immediately.
***

Reply via email to