Hi all,
We run SolrCloud 9.6.1 with 63 shards , with one product category per
shard.We have added three 2048-dimensional KNN vector fields and want to
perform per-shard newSearcher/firstSearcher warmup using real,
category-relevant KNN queries. However, each warmup query contains a
literal float vector (~30KB+), so a shard's warmup configuration can
quickly exceed ZooKeeper's 1MB znode limit when stored in the configset.
What we have tried so far:
1.Splitting the warmup file into multiple chunks under 1MB using
multiple xi:includes. This works, but the files still reside in ZooKeeper,
so any warmup update requires re-pushing the configset.
2.Using xi:include with an absolute local file path. This does not work
because XInclude resolution goes through the ZK-backed configset resource
loader rather than reading arbitrary files from the local filesystem.
3.Our current approach is a custom AbstractSolrEventListener that uses
plain java.io to read a local, per-shard XML file at warmup time, keyed by
${solr.core.name}. This bypasses the configset loader. We also keep a small
fallback warmup list in the configset itself.
We have a few questions:
1.Is there a more idiomatic Solr configuration mechanism for having a
warmup listener read per-node or per-core local files, without implementing
a custom listener?
2.Is there a known pattern for per-shard or category-specific warmup in a
manually sharded collection?
3.For KNN warmup, is embedding the literal query vector in the warmup query
the normal approach? Or is there another recommended way to warm
HNSW-related structures, for example by referencing a stored document's
vector by ID?
Our main goal is to keep the warmup data local to each shard/core, since
each shard represents a different product category and does not need
vectors from other categories.
Happy to share the custom listener code if it is useful.
Thanks!
Sunny
