Michael's work was indeed never merged. This approach of searching the IndexWriter buffer is tempting on paper, but I worry that it would come with lots of asterisks and be hard to use in practice. - Terms and points are not sorted in the IW buffer, so some queries like range queries would be unsupported or run linear scans. - Postings don't have index statistics, skip data or impacts in the IW buffer, so we would need to cache more data (at the expense of higher memory usage) or run linear scans. - IndexWriter could not release heap from a DWPT if a search is running on this DWPT.
Letting the caller decide whether to wait for full-flush merges or not on reopen would require Lucene to track full-flush merges a bit more carefully but it sounds doable. This is something we would probably take advantage of in Elasticsearch as we sometimes reopen indexes for internal reasons, where we don't care about waiting for full-flush merges (while we still want to wait for full-flush merges for user-facing searches). On Tue, Jul 30, 2024 at 9:42 PM David Smiley <dsmi...@apache.org> wrote: > On Mon, Jul 29, 2024 at 3:51 PM Michael Froh <msf...@gmail.com> wrote: > > > > Hi David, > > > > Great meeting you at Buzzwords last month! > > Nice seeing you too! > > ... > > Adding an overload to IndexWriter.getReader would be pretty easy, but > that method is package-private. The hairier part probably involves deciding > which open/openIfChanged methods to overload in DirectoryReader, since I > think that's where the parameter would ultimately need to be exposed to > users. > > Yes. > > Admittedly this isn't something I'm actively seeking right now. I was > motivated by Solr's code that has a "realtime" searcher distinction > internally, in which we try to open a searcher ASAP.. However I > discovered that in practice there is no practical distinction since > the regular searcher is almost always the same, thus opening searchers > sometimes faster isn't worth it. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org > > -- Adrien
