Hi Steven,

Great question!  I'm so glad to hear your app is providing consistent
pagination :)  I've long felt Lucene (with NRT segment replication) could
do a great job at this, yet so few apps manage to implement it.  Every time
I interact with a search engine and go to the next page it irks me that I
might be missing some results...

First off, the point-in-time IndexSearchers are keyed into the
SearcherLifetimeManager by their underlying IndexReader.getVersion(), which
returns the long value from the underlying SegmentInfos.getVersion().

This is good news because it means all of your replicas will see the same
long version mapping to the same point-in-time view of the index,
even across replicas, since that same SegmentInfos is sent to all replicas
by the primary node.

Second, each of your replicas should simply assume that every point-in-time
IndexSearcher may be used at any time by an incoming search request, and
enroll all refreshed IndexSearchers into the local
SearcherLifetimeManager.  This way, no matter where the followon requests
go, that replica will have that IndexSearcher version.   This is not as
costly as it sounds because a refreshed IndexSearcher will in general share
nearly all of its segments with the prior one(s).

This requires a periodic refresh schedule, and all replicas should quickly
refresh when the primary publishes a new point-in-time SegmentInfos.

There is some small risk if replicas do not refresh consistently around
the same time, and page 2 for a query goes to a replica that has not yet
refreshed.  This ought to be rare, since it'd mean a human loaded page 1
from a replica that had already refreshed, consumed the results, then
clicked on page 2, and by then replicas should (typically) all have
refreshed.  When it happens, you could either have the query wait for the
refresh to completely (somewhat dangerous since such queries could pile up
if something is seriously wrong with that node and its refreshing is
sluggish), or, simply retry the query to another replica: eventually it
will find a replica that has the point-in-time IndexSearcher already

Mike McCandless


On Wed, Dec 13, 2023 at 4:47 PM Steven Schlansker <
stevenschlans...@gmail.com> wrote:

> Hi lucene-users,
> We use the lucene-replicator to have a single indexing node push commits
> and NRT updates
> to a set of replicas.
> Currently, each replica has the full dataset - there is no sharding.
> We use a SearcherLifetimeManager to try to provide consistent pagination
> over results.
> So when we present the first page of results, we return the result of
> `record(IndexSearcher)` to the client,
> with the expectation that at a later time (but not too much later) they
> might request page 2 of results with version X.
> This works fine for a single instance, since the SearcherLifetimeManager
> keeps the remembered version around.
> However, with multiple instances, this doesn't seem to work at all -
> your first request goes to replica A, who calls `record(searcher) -> X`.
> The second request likely goes to a different instance B,
> whose lifetime manager never saw a call to `record` at all - so the
> `acquire(X)` fails and returns null.
> Surely there must be a way to solve this -
> how do you implement consistent versioned searching like
> SearcherLifetimeManager, but with multiple Lucene replicas
> who otherwise do not coordinate about which NRT versions get opened or
> recorded?
> Thanks for any advice,
> Steven
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
> For additional commands, e-mail: java-user-h...@lucene.apache.org

Reply via email to