On 2026/10/16 03:20:25 Enlai Li wrote:
> Hi Eric,
>
> Thanks for looking at this.
>
> The cluster reload you describe comes from touchConfDir(). The current
> Schema API writes the managed schema and then bumps the configset znode,
> and the listener on that znode reloads every core using the configset.
>
> This change does not do that for a request containing only add-field,
> add-dynamic-field and add-copy-field. It calls persistManagedSchema(),
> which updates the managed-schema znode and leaves the configset znode
> alone. The schema file in ZooKeeper is updated. Replicas already watch the
> managed-schema znode in ZkIndexSchemaReader and install it with
> setLatestSchema(), without a reload. That is the path schemaless mode uses,
> so the replica that received the request and the others behave the same way.
>
> One change from my first mail: I am no longer opening a searcher on every
> replica. In SolrCloud, queries see the new field after the next commit or
> soft commit. Opening one only on the replica that handled the request would
> make a distributed query succeed or fail depending on which replica it
> hits. Standalone still opens a searcher before responding, which keeps
> SOLR-13795.
>
> On similarity: the IndexWriter keeps the Similarity instance it was given
> when it was constructed. The default SchemaSimilarityFactory returns a
> PerFieldSimilarityWrapper that reads core.getLatestSchema() for each field,
> so a new field of an existing field type picks up that type's similarity
> without a new writer. A custom SimilarityFactory that snapshots its
> similarities at construction time would not see the new field until the
> writer is replaced. This change does not cover that case. Delete, replace
> and mixed requests still reload.
>
> The change is one commit on schema-api-additive-no-reload:
>
> https://github.com/linslee75/solr/commit/d692a66dbf
>
> TestSchemaChangeDuringMerge holds a merge open and checks that an update
> submitted during add-field returns. On main the same test fails, because
> add-field reloads the core and the update stays blocked until the merge
> finishes. TestManagedSchemaAPI checks the cluster case: after add-field,
> add-dynamic-field and add-copy-field, no replica's core uniqueId changes.
>
> Thanks,
> Li Enlai
>
> This message was drafted with AI assistance and reviewed by me.
>
> On Fri, Oct 16, 2026 at 1:31 AM Eric Pugh <[email protected]> wrote:
>
> >
> >
> > On 2026-10-12 12:01 UTC Enlai Li wrote:
> > > Hi,
> > >
> > > Schema API add-field, add-dynamic-field, and add-copy-field currently
> > > persist the managed schema and reload every core using that configset.
> > > Reload calls newIndexWriter(), which closes the old IndexWriter and waits
> > > for running merges while holding the writer lock. Under continuous
> > > indexing this blocks updates until those merges finish. Vector fields
> > make
> > > the merges expensive, so the stall can last for minutes.
> > >
> > > Schemaless mode already adds fields without a reload: persist the managed
> > > schema and install it with setLatestSchema(). Writes see the new field
> > > because the IndexWriter reads the core's latest schema. I would like the
> > > same for these three Schema API operations only. Delete and replace
> > > operations would still reload.
> > >
> > > One difference from schemaless: after installing the schema, open a new
> > > searcher without committing, so queries can use the new field
> > immediately.
> > > The current searcher keeps the old schema.
> > >
> > > Any objection to this approach?
> > >
> > > Thanks,
> > > Li Enlai
> > >
> > > This message was drafted with AI assistance and reviewed by me.
> > >
> >
> > Hi Li,
> >
> > I am not an expert in this area, but wanted to chime in to give some
> > feedback.
> >
> > The idea is simpler to do in a single node setup, but if we are
> > distributed then an issue may be that a Schema API change propagates to
> > other replicas via a ZK watch on the configset, and that watch handler
> > currently does an unconditional full core.reload() whenever the schema
> > version bumps — it doesn't distinguish additive from destructive changes.
> > So unless that path also gets updated, this would only skip the reload on
> > the node that received the API call; every other replica in the collection
> > would still reload and stall the same way. Given the problem is
> > specifically about continuous indexing across a cluster, that seems like
> > it'd need to be part of the change too.
> >
> > Smaller thing: the IndexWriter's analyzer is re-fetched from the schema on
> > every lookup, so it naturally picks up new fields — but Similarity is
> > captured once at writer-construction time, not re-fetched. Probably fine
> > for most setups, but worth a second look if anyone's using custom per-field
> > similarity. Or something that might need to trigger the full .reload?
> >
> > I like the idea of making Solr more robust and friendlier to changing
> > schemas. Lots of cool use cases where you are getting new fields added to
> > your setup as new data comes in!
> >
> > Thanks,
> > Eric
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [email protected]
> > For additional commands, e-mail: [email protected]
> >
> >
>
So.. I guess a little bit is "what does this all mean?" What are the
ramifications of this change? One thing that helps me sometimes is a bash/curl
based reproduction script that shows how things work from a end user
perspective.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
