I don't know where you came from, Serhiy, but I'm super appreciative of your efforts to improve Solr! BTW I hope you can get enough code reviews but that's a limiting factor that will require some patience. Personally I need to dedicate more time to my day job than in some previous weeks.
On Fri, Jul 17, 2026 at 11:00 AM Serhiy Bzhezytskyy < [email protected]> wrote: > Hello All, > Following up on David's "State of Lucene Index Sorting" report from a few > weeks back. That report reads as a roadmap — a handful of related tickets > that together would make index sorting first-class in Solr — so I spent > time working through them end to end. I have local, tested implementations > for the open pieces and I'd like to bring them to the list before opening > PRs, since several sit in areas others own (particularly Christine's #313). > > I'll lay out how I see the puzzle, what I've built, and where I'd like > direction / to collaborate. > > *The picture (from David's report)* > > Index sorting works today, but only through the indirect > `SortingMergePolicyFactory` — a merge policy that does no merging and > exists only to carry a `Sort` to `IndexWriterConfig.setIndexSort()`. > > The report flagged five gaps; working through them surfaced two adjacent > pieces (SOLR-15390, SOLR-17310). > > The full map as I now see it: > - SOLR-13681 — direct <indexSort> config — Open, Christine's draft #313 > - SOLR-9108 — improve sort config (overlaps 13681) — Open > - SOLR-12230 — deprecate SortingMergePolicy — Open > - SOLR-12239 — re-sort an existing collection — Open, my PR #4644 > - SOLR-17170 — nested/block-join sorting — Resolved (David) > - SOLR-15390 — modernize the early-termination collector (adjacent) — Patch > Available > - SOLR-17310 — configurable leaf sorter (adjacent) — Open, Wei Wang's #2477 > (stale-bot auto-closed, not rejected) > > *What I've implemented and verified locally* > > 1. SOLR-13681 — direct <indexSort> config. > A first-class <indexConfig><indexSort>timestamp > desc</indexSort></indexConfig>, parsed with the same SortSpecParsing the > query sort uses, replacing the merge-policy indirection. This reimplements > Christine's #313 approach on current main (the draft predates `MapWriter` > and David's SOLR-17170 parent-field work, so it needed reworking) and > composes with the nested-docs parent field. Christine — this is your lane > and I'm not looking to step on #313. I'd rather hand this over or work with > you on it — or simply absorb any of it into #313 as you see fit, with or > without me. I just found it was the keystone the other pieces depend on, > and its long-standing blocker ("what about existing collections?") is > answered by SOLR-12239. > > 2. SOLR-12239 — re-sort an existing collection (PR #4644, already up). > A RESORTINDEX core-admin action (v1 + v2) using LUCENE-9484 > (SortingCodecReader + addIndexes) to sort pre-existing segments without > a full reindex — removing the "sort is immutable, must reindex" limitation. > It reads the configured <indexSort> as its target, so the workflow is: set > <indexSort> → run RESORTINDEX once. > > 3. SOLR-17310 — configurable leaf sorter (<segmentSort>). > Builds on Wei Wang's #2477 (credited). Exposes Lucene's `setLeafSorter` for > *between-segment* ordering, distinct from index sort's *within-segment* > order. Generalized beyond the time-only version to also accept a > numeric-field spec, since setLeafSorter takes any Comparator<LeafReader>. > > 4. SOLR-15390 — modernize the deprecated collector. > Solr's segmentTerminateEarly rides a > @Deprecated EarlyTerminatingSortingCollector; I swapped it for Lucene's > native TopFieldCollector.isEarlyTerminated(), preserving the > `segmentTerminatedEarly` response semantics. The ticket's broader ask is > deprecating the segmentTerminateEarly param in favor of `minExactCount` — I > did the internal modernization but left the param-deprecation as a separate > decision, since it's user-facing. > > 5. SOLR-12230 — deprecate SortingMergePolicy(Factory). > Once <indexSort> exists as the replacement, the old carrier can be > deprecated. Done as the last step, with a Solr-11 upgrade note. > > Everything is tested (unit + SolrCloud; ~40 tests) and passes errorprone / > check / ref-guide link checks. > > *Two design questions I'd like the list's view on* > > - Config shape: <indexSort> (within-segment) and <segmentSort> > (between-segment) as *separate* <indexConfig> elements, vs. the > <indexSorters> unified-element idea from SOLR-17310. My reading > of precedent leans separate — Lucene keeps `setIndexSort`/`setLeafSorter` > as distinct setters, and ES/OpenSearch expose only the within-segment axis > (index.sort.*) with no leaf-sorter equivalent — but I don't feel strongly > and want to hear opinions. > - segmentTerminateEarly vs minExactCount: actually deprecate the param > (SOLR-15390's literal ask), or just modernize the internals as I've done? > > What I'm asking is mainly direction and collaboration — especially with > Christine on 13681, since that's the keystone and her active work. If > there's interest, I'll open the pieces as separate per-ticket PRs (#4644 > is already up). OR open PR with all changes to see full picture. Happy to > adjust any of it. > > Thanks, > Serhiy > > On Wed, 15 Jul 2026 at 20:01, David Smiley <[email protected]> wrote: > > > I'm working on a project / idea that will require it. I'll share more > > about that later when appropriate, but it's too early now. > > > > At least the current state isn't bad. It works, especially with expanded > > functionality extending to schemas/use-cases with nested docs: SOLR-17170 > > -- fully merged back to 9.x. There's a usability issue that I hope > > Christine will prioritize improving. > > > > I've heard Adrien and others extol the virtues/benefits of index sorting > > but haven't yet had deployed it to share. I will when I can. > > > > On Wed, Jul 15, 2026 at 8:06 AM Jason Gerlowski <[email protected]> > > wrote: > > > > > Curious David - are you considering using index sorting for a > > > use-case, or have any experience with it to share in terms of > > > speedups, etc? Or more just something that struck your fancy? > > > > > > I remember index-sorting being something that came up in a Community > > > over Code "Birds of a Feather" session (in, I believe, Halifax?), as > > > one of the things Lucene supported that Solr should really consider > > > enabling out of the box for performance benefits. But I can't > > > remember the details of that discussion very well... > > > > > > Best, > > > > > > Jason > > > > > > On Thu, Jul 2, 2026 at 12:20 PM David Smiley <[email protected]> > wrote: > > > > > > > > I started investigating the current state of Lucene's index sorting > > > support > > > > in Solr. I had Claude Opus write a report for me. Rather than hoard > > it > > > to > > > > myself, I'm sharing with everyone in case others are wondering what's > > up > > > as > > > > well. > > > > > > > > > > > > Background > > > > ---------- > > > > > > > > Lucene has supported index-level sorting since LUCENE-6766 (Lucene > > 6.2), > > > > where segments are internally sorted by a configurable field order at > > > > flush/merge time. This enables significant query-time optimizations > -- > > > > when the query's sort matches the index sort, Lucene can skip entire > > > > segments or terminate collection early. > > > > > > > > > > > > Current Solr Support > > > > -------------------- > > > > > > > > Solr does support index sorting today, but through an indirect > > mechanism: > > > > > > > > Configuration is done via SortingMergePolicyFactory in > solrconfig.xml: > > > > > > > > <mergePolicyFactory > > > > class="org.apache.solr.index.SortingMergePolicyFactory"> > > > > <str name="sort">timestamp desc</str> > > > > <str name="wrapped.prefix">inner</str> > > > > <str > > > > > name="inner.class">org.apache.solr.index.TieredMergePolicyFactory</str> > > > > </mergePolicyFactory> > > > > > > > > Internally, SortingMergePolicy is a FilterMergePolicy that does > nothing > > > > merge-policy-related -- it simply holds a Sort object. > SolrIndexConfig > > > > then has a special instanceof check that extracts this Sort and calls > > > > IndexWriterConfig.setIndexSort(). The class itself has a TODO comment > > > > acknowledging this is a workaround: "remove this and add indexSort > > > > specification directly to solrconfig.xml?" > > > > > > > > Query-side integration exists via the "segmentTerminateEarly" query > > > > parameter, which wraps the collector in an > > > EarlyTerminatingSortingCollector. > > > > Note that this collector is @Deprecated -- modern Lucene's > > > TopFieldCollector > > > > handles early termination natively when it detects sorted segments. > > > > > > > > The /admin/segments API (with coreInfo=true) exposes the indexSort > > > > configuration and per-segment sort info. > > > > > > > > AtomicUpdateDocumentMerger correctly detects fields used for index > > > sorting > > > > and prevents DocValues-only updates on them (a Lucene limitation). > > > > > > > > > > > > Open Issues > > > > ----------- > > > > > > > > Several open JIRA issues relate to this area. > > > > > > > > SOLR-9108: Improve how index time sorting is configured > > > > https://issues.apache.org/jira/browse/SOLR-9108 > > > > > > > > Filed by Mike McCandless in 2016 right after LUCENE-6766. Proposes > > > > configuring index sort directly in solrconfig.xml alongside other > > > > IndexWriter settings rather than piggybacking on the merge policy. > > > > > > > > SOLR-13681: Make Lucene's index sorting directly configurable in Solr > > > > https://issues.apache.org/jira/browse/SOLR-13681 > > > > > > > > Filed by Christine Poerschke in 2019. Has a draft PR (#313) that > adds > > > > a direct <indexSort> config element to solrconfig.xml. The PR has > > been > > > > stalled since 2021; the main open question is what should happen > when > > > > index sorting is enabled on an existing collection that already has > > > > unsorted segments. Duplicates SOLR-12230 (deprecate > > > SortingMergePolicy). > > > > > > > > SOLR-12239: Enabling index sorting causes CorruptIndexException > > > > https://issues.apache.org/jira/browse/SOLR-12239 > > > > > > > > When index sorting is enabled on an existing collection with > unsorted > > > > segments, reloading throws: "segment not sorted with > indexSort=null". > > > > The current workaround is to delete all data and reindex from > > scratch. > > > > Notably, the related LUCENE-9484 ("Allow index sorting to happen > > after > > > > the fact") was fixed in Lucene 9.0, which allows merging unsorted > > > > segments into sorted ones retroactively. Solr has not wired this > up. > > > > > > > > SOLR-17170: Support Blocks in Index Sorting > > > > https://issues.apache.org/jira/browse/SOLR-17170 > > > > > > > > Lucene 9.10+ supports block-aware presort during index sorting (via > > > > Lucene PR #12829). This is critical for nested/block-join > documents. > > > > No Solr-side work has been done. > > > > > > > > > > > > Summary > > > > ------- > > > > > > > > Index sorting in Solr works for simple (non-nested) use cases via the > > > > SortingMergePolicyFactory, but the implementation is showing its age: > > > > > > > > - Configuration is indirect and hacky (merge policy as a Sort > carrier) > > > > - Cannot be safely enabled on existing collections without full > > reindex, > > > > despite Lucene having solved this at the engine level since 9.0 > > > > - Incompatible with nested/block-join documents on Solr 10+ > > > > - The query-side early termination collector is deprecated > > > > - A draft PR for direct configuration has been stalled since 2021 > > > > > > > > ~ David Smiley > > > > Apache Lucene/Solr Search Developer > > > > http://www.linkedin.com/in/davidwsmiley > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: [email protected] > > > For additional commands, e-mail: [email protected] > > > > > > > > >
