> On Feb 26, 2025, at 2:53 PM, Marc Davenport
> <madavenp...@cargurus.com.INVALID> wrote:
>
> Hello,
> Our current search solution is a pretty big monolith running on pretty
> beefy EC2 instances. Every node is responsible for indexing and serving
> queries. We want to start decomposing our service and are starting with
> separating the indexing and query handling responsibilities.
We run a probably comparatively small but otherwise similar installation, using
Google Kubernetes instances. We just use a persistent disk instead of an
elastic store, but
also would consider using something like S3 in the future.
> I'm in the research phases now trying to collect any prior art I can. The
> rough sketch is to implement the NRT two replication node classes on their
> respective services and use S3 as a distribution point for the segment
> files. I'm still debating if there should be some direct knowledge of the
> replicas in the primary node.
We tried to avoid the primary keeping any durable state about replicas,
as replicas tend to disappear for any or no reason in a cloud environment.
A specific example issue we ran into: we disable the 'waitForAllRemotesToClose'
step entirely
https://github.com/apache/lucene/pull/11822
> Or if the primary node can just churn away
> creating base indexes and updates and publish to a queue when it produces a
> new set of segments. Then the replicas are then free to pick up the latest
> index as they spin up and subscribe to changes for it. It seems like having
> the indexer being responsible for also communicating with the replicas
> would be double duty for that system. I'd love to hear other experiences
> if people can share them or point to writings about them they read when
> designing their systems.
In our case, the primary keeps the latest CopyState that any replica should
take in memory.
Replicas call a HTTP api in an infinite loop, passing in their current version,
and asking if any newer version is available.
If the replica is behind, the primary gives it the current CopyState NRT point
immediately.
If the replica is caught up, we hold the request in a long-poll for just short
of our http timeout, waiting for
a new version to become available, otherwise return "no update for now, try
again".
Once the replica receives an updated CopyState, it feeds it into the
ReplicaNode with newNRTPoint which starts the file copy
There's a bit of magic we devised internally around retaining the CopyState
reference count, as this is stateful and the HTTP replication is mostly
stateless.
We keep this simple by having only a single primary node at any given time.
Your idea of using a queue instead is interesting but not something we
extensively looked at :)
> I've looked at nrtsearch from yelp and they seem to let the primary node
> have direct knowledge of the replicas. That makes sense since it is based
> on McCandless's LuceneServer.
>
> I know that Amazon internally uses Lucene and has indexing separated from
> query nodes and that they re-index and publish completely new indexes with
> every release to prod. I've been watching what I can of the great videos of
> Sokolov, McCandless, & Froh etc. But they don't show much behind the
> curtain (understandably) about the details of keeping things in sync. If
> someone does know of a publicly available video or resource that describes
> this, I'd love to see it.
Unfortunately our journey with the LuceneServer was basically informed by the
code and blog posts you've likely already seen -
and a bit of help from this mailing list - otherwise, there's not a ton of
information out there or evidence of widespread use.
That said, we've been very happy with our setup, despite needing to put in a
fair amount of elbow grease and low level details to get things working for us.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org
