Thanks David for the feedback — it really charged my enthusiasm and keeps the garage door open for more and more improvements (hope not breaking things 🙂 ).
On the pace — the quick reviews were the major fuel for me during onboarding, more than I expected. Now I'm ready to adapt to any pace that works. PR "index-sorting-all-in-one": github.com/apache/solr/pull/4648 On Tue, 21 Jul 2026 at 07:05, David Smiley <[email protected]> wrote: > I don't know where you came from, Serhiy, but I'm super appreciative of > your efforts to improve Solr! > BTW I hope you can get enough code reviews but that's a limiting factor > that will require some patience. > Personally I need to dedicate more time to my day job than in some previous > weeks. > > On Fri, Jul 17, 2026 at 11:00 AM Serhiy Bzhezytskyy < > [email protected]> wrote: > > > Hello All, > > Following up on David's "State of Lucene Index Sorting" report from a few > > weeks back. That report reads as a roadmap — a handful of related tickets > > that together would make index sorting first-class in Solr — so I spent > > time working through them end to end. I have local, tested > implementations > > for the open pieces and I'd like to bring them to the list before opening > > PRs, since several sit in areas others own (particularly Christine's > #313). > > > > I'll lay out how I see the puzzle, what I've built, and where I'd like > > direction / to collaborate. > > > > *The picture (from David's report)* > > > > Index sorting works today, but only through the indirect > > `SortingMergePolicyFactory` — a merge policy that does no merging and > > exists only to carry a `Sort` to `IndexWriterConfig.setIndexSort()`. > > > > The report flagged five gaps; working through them surfaced two adjacent > > pieces (SOLR-15390, SOLR-17310). > > > > The full map as I now see it: > > - SOLR-13681 — direct <indexSort> config — Open, Christine's draft #313 > > - SOLR-9108 — improve sort config (overlaps 13681) — Open > > - SOLR-12230 — deprecate SortingMergePolicy — Open > > - SOLR-12239 — re-sort an existing collection — Open, my PR #4644 > > - SOLR-17170 — nested/block-join sorting — Resolved (David) > > - SOLR-15390 — modernize the early-termination collector (adjacent) — > Patch > > Available > > - SOLR-17310 — configurable leaf sorter (adjacent) — Open, Wei Wang's > #2477 > > (stale-bot auto-closed, not rejected) > > > > *What I've implemented and verified locally* > > > > 1. SOLR-13681 — direct <indexSort> config. > > A first-class <indexConfig><indexSort>timestamp > > desc</indexSort></indexConfig>, parsed with the same SortSpecParsing the > > query sort uses, replacing the merge-policy indirection. This > reimplements > > Christine's #313 approach on current main (the draft predates `MapWriter` > > and David's SOLR-17170 parent-field work, so it needed reworking) and > > composes with the nested-docs parent field. Christine — this is your lane > > and I'm not looking to step on #313. I'd rather hand this over or work > with > > you on it — or simply absorb any of it into #313 as you see fit, with or > > without me. I just found it was the keystone the other pieces depend on, > > and its long-standing blocker ("what about existing collections?") is > > answered by SOLR-12239. > > > > 2. SOLR-12239 — re-sort an existing collection (PR #4644, already up). > > A RESORTINDEX core-admin action (v1 + v2) using LUCENE-9484 > > (SortingCodecReader + addIndexes) to sort pre-existing segments without > > a full reindex — removing the "sort is immutable, must reindex" > limitation. > > It reads the configured <indexSort> as its target, so the workflow is: > set > > <indexSort> → run RESORTINDEX once. > > > > 3. SOLR-17310 — configurable leaf sorter (<segmentSort>). > > Builds on Wei Wang's #2477 (credited). Exposes Lucene's `setLeafSorter` > for > > *between-segment* ordering, distinct from index sort's *within-segment* > > order. Generalized beyond the time-only version to also accept a > > numeric-field spec, since setLeafSorter takes any Comparator<LeafReader>. > > > > 4. SOLR-15390 — modernize the deprecated collector. > > Solr's segmentTerminateEarly rides a > > @Deprecated EarlyTerminatingSortingCollector; I swapped it for Lucene's > > native TopFieldCollector.isEarlyTerminated(), preserving the > > `segmentTerminatedEarly` response semantics. The ticket's broader ask is > > deprecating the segmentTerminateEarly param in favor of `minExactCount` > — I > > did the internal modernization but left the param-deprecation as a > separate > > decision, since it's user-facing. > > > > 5. SOLR-12230 — deprecate SortingMergePolicy(Factory). > > Once <indexSort> exists as the replacement, the old carrier can be > > deprecated. Done as the last step, with a Solr-11 upgrade note. > > > > Everything is tested (unit + SolrCloud; ~40 tests) and passes errorprone > / > > check / ref-guide link checks. > > > > *Two design questions I'd like the list's view on* > > > > - Config shape: <indexSort> (within-segment) and <segmentSort> > > (between-segment) as *separate* <indexConfig> elements, vs. the > > <indexSorters> unified-element idea from SOLR-17310. My reading > > of precedent leans separate — Lucene keeps `setIndexSort`/`setLeafSorter` > > as distinct setters, and ES/OpenSearch expose only the within-segment > axis > > (index.sort.*) with no leaf-sorter equivalent — but I don't feel strongly > > and want to hear opinions. > > - segmentTerminateEarly vs minExactCount: actually deprecate the param > > (SOLR-15390's literal ask), or just modernize the internals as I've done? > > > > What I'm asking is mainly direction and collaboration — especially with > > Christine on 13681, since that's the keystone and her active work. If > > there's interest, I'll open the pieces as separate per-ticket PRs (#4644 > > is already up). OR open PR with all changes to see full picture. Happy to > > adjust any of it. > > > > Thanks, > > Serhiy > > > > On Wed, 15 Jul 2026 at 20:01, David Smiley <[email protected]> wrote: > > > > > I'm working on a project / idea that will require it. I'll share more > > > about that later when appropriate, but it's too early now. > > > > > > At least the current state isn't bad. It works, especially with > expanded > > > functionality extending to schemas/use-cases with nested docs: > SOLR-17170 > > > -- fully merged back to 9.x. There's a usability issue that I hope > > > Christine will prioritize improving. > > > > > > I've heard Adrien and others extol the virtues/benefits of index > sorting > > > but haven't yet had deployed it to share. I will when I can. > > > > > > On Wed, Jul 15, 2026 at 8:06 AM Jason Gerlowski <[email protected] > > > > > wrote: > > > > > > > Curious David - are you considering using index sorting for a > > > > use-case, or have any experience with it to share in terms of > > > > speedups, etc? Or more just something that struck your fancy? > > > > > > > > I remember index-sorting being something that came up in a Community > > > > over Code "Birds of a Feather" session (in, I believe, Halifax?), as > > > > one of the things Lucene supported that Solr should really consider > > > > enabling out of the box for performance benefits. But I can't > > > > remember the details of that discussion very well... > > > > > > > > Best, > > > > > > > > Jason > > > > > > > > On Thu, Jul 2, 2026 at 12:20 PM David Smiley <[email protected]> > > wrote: > > > > > > > > > > I started investigating the current state of Lucene's index sorting > > > > support > > > > > in Solr. I had Claude Opus write a report for me. Rather than > hoard > > > it > > > > to > > > > > myself, I'm sharing with everyone in case others are wondering > what's > > > up > > > > as > > > > > well. > > > > > > > > > > > > > > > Background > > > > > ---------- > > > > > > > > > > Lucene has supported index-level sorting since LUCENE-6766 (Lucene > > > 6.2), > > > > > where segments are internally sorted by a configurable field order > at > > > > > flush/merge time. This enables significant query-time optimizations > > -- > > > > > when the query's sort matches the index sort, Lucene can skip > entire > > > > > segments or terminate collection early. > > > > > > > > > > > > > > > Current Solr Support > > > > > -------------------- > > > > > > > > > > Solr does support index sorting today, but through an indirect > > > mechanism: > > > > > > > > > > Configuration is done via SortingMergePolicyFactory in > > solrconfig.xml: > > > > > > > > > > <mergePolicyFactory > > > > > class="org.apache.solr.index.SortingMergePolicyFactory"> > > > > > <str name="sort">timestamp desc</str> > > > > > <str name="wrapped.prefix">inner</str> > > > > > <str > > > > > > > name="inner.class">org.apache.solr.index.TieredMergePolicyFactory</str> > > > > > </mergePolicyFactory> > > > > > > > > > > Internally, SortingMergePolicy is a FilterMergePolicy that does > > nothing > > > > > merge-policy-related -- it simply holds a Sort object. > > SolrIndexConfig > > > > > then has a special instanceof check that extracts this Sort and > calls > > > > > IndexWriterConfig.setIndexSort(). The class itself has a TODO > comment > > > > > acknowledging this is a workaround: "remove this and add indexSort > > > > > specification directly to solrconfig.xml?" > > > > > > > > > > Query-side integration exists via the "segmentTerminateEarly" query > > > > > parameter, which wraps the collector in an > > > > EarlyTerminatingSortingCollector. > > > > > Note that this collector is @Deprecated -- modern Lucene's > > > > TopFieldCollector > > > > > handles early termination natively when it detects sorted segments. > > > > > > > > > > The /admin/segments API (with coreInfo=true) exposes the indexSort > > > > > configuration and per-segment sort info. > > > > > > > > > > AtomicUpdateDocumentMerger correctly detects fields used for index > > > > sorting > > > > > and prevents DocValues-only updates on them (a Lucene limitation). > > > > > > > > > > > > > > > Open Issues > > > > > ----------- > > > > > > > > > > Several open JIRA issues relate to this area. > > > > > > > > > > SOLR-9108: Improve how index time sorting is configured > > > > > https://issues.apache.org/jira/browse/SOLR-9108 > > > > > > > > > > Filed by Mike McCandless in 2016 right after LUCENE-6766. > Proposes > > > > > configuring index sort directly in solrconfig.xml alongside other > > > > > IndexWriter settings rather than piggybacking on the merge > policy. > > > > > > > > > > SOLR-13681: Make Lucene's index sorting directly configurable in > Solr > > > > > https://issues.apache.org/jira/browse/SOLR-13681 > > > > > > > > > > Filed by Christine Poerschke in 2019. Has a draft PR (#313) that > > adds > > > > > a direct <indexSort> config element to solrconfig.xml. The PR has > > > been > > > > > stalled since 2021; the main open question is what should happen > > when > > > > > index sorting is enabled on an existing collection that already > has > > > > > unsorted segments. Duplicates SOLR-12230 (deprecate > > > > SortingMergePolicy). > > > > > > > > > > SOLR-12239: Enabling index sorting causes CorruptIndexException > > > > > https://issues.apache.org/jira/browse/SOLR-12239 > > > > > > > > > > When index sorting is enabled on an existing collection with > > unsorted > > > > > segments, reloading throws: "segment not sorted with > > indexSort=null". > > > > > The current workaround is to delete all data and reindex from > > > scratch. > > > > > Notably, the related LUCENE-9484 ("Allow index sorting to happen > > > after > > > > > the fact") was fixed in Lucene 9.0, which allows merging unsorted > > > > > segments into sorted ones retroactively. Solr has not wired this > > up. > > > > > > > > > > SOLR-17170: Support Blocks in Index Sorting > > > > > https://issues.apache.org/jira/browse/SOLR-17170 > > > > > > > > > > Lucene 9.10+ supports block-aware presort during index sorting > (via > > > > > Lucene PR #12829). This is critical for nested/block-join > > documents. > > > > > No Solr-side work has been done. > > > > > > > > > > > > > > > Summary > > > > > ------- > > > > > > > > > > Index sorting in Solr works for simple (non-nested) use cases via > the > > > > > SortingMergePolicyFactory, but the implementation is showing its > age: > > > > > > > > > > - Configuration is indirect and hacky (merge policy as a Sort > > carrier) > > > > > - Cannot be safely enabled on existing collections without full > > > reindex, > > > > > despite Lucene having solved this at the engine level since 9.0 > > > > > - Incompatible with nested/block-join documents on Solr 10+ > > > > > - The query-side early termination collector is deprecated > > > > > - A draft PR for direct configuration has been stalled since 2021 > > > > > > > > > > ~ David Smiley > > > > > Apache Lucene/Solr Search Developer > > > > > http://www.linkedin.com/in/davidwsmiley > > > > > > > > --------------------------------------------------------------------- > > > > To unsubscribe, e-mail: [email protected] > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > >
