> @Sarthak - I see the term pre-copy all over LuceneServer & nrtSearch but I > haven't been able to distinguish the term from just "copy". Does the "pre" > simply refer to the fact that the transfer of bits is happening before the > replica starts to serve queries from that segment? I feel like I've been > missing something when I read that code. > Precopy refers to copying merged segments to replicas before any of the replicas start using the segments. Once the precopy is complete, all replicas can start using the new segments at the same time.
On Mon, Mar 3, 2025 at 2:00 PM Michael Froh <msf...@gmail.com> wrote: > On Sun, Mar 2, 2025 at 7:21 AM Marc Davenport > <madavenp...@cargurus.com.invalid> wrote: > > > > > @Michael - That second simpler architecture is very similar to what we > are > > considering; With the exception of a queue for announcing new > > segments rather than a polling process. It is good to know that it's a > > reasonable outline. You were very latency sensitive. Is there anything > > you can share around the most important specs of your containers, pods, > or > > even nodes? While we run on these EC2 instances, there is a push to get > us > > into k8s. Did you have any issues as you migrated from one version of > > lucene to another. I'm concerned that our current deployments only allow > > one version of the software in production at any one time. > > > > > From what I remember, we did see a bit of noisy-neighbor behavior running > two searchers in separate containers on the same instance. I don't think > we ever found a concrete reason (at least while I was there). We gave up > and ended up using smaller instances instead of putting multiple searchers > on the same instance. > > For migration between different Lucene versions (or even versions of our > software, which might change indexing behavior), we used the process > described in > > https://careersatdoordash.com/blog/introducing-doordashs-in-house-search-engine/ > in the section "Tenant Isolation and Search Stacks". Essentially, bring up > a new indexer fleet, build a new index, then start bringing up new > searchers that replicate from the new indexers. >
