[
https://issues.apache.org/jira/browse/SOLR-18304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SOLR-18304:
----------------------------------
Labels: pull-request-available (was: )
> Fix collapse on String performance regression due to Lucene upgrade
> -------------------------------------------------------------------
>
> Key: SOLR-18304
> URL: https://issues.apache.org/jira/browse/SOLR-18304
> Project: Solr
> Issue Type: Improvement
> Components: search
> Affects Versions: 8.9
> Reporter: David Smiley
> Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Migrating to {{Lucene90DocValuesProducer}} in Solr 9 revealed a significant
> performance regression in collapse queries sorted by string fields, due to
> the extra overhead of LZ4 decompression. Solr 9’s collapse implementation
> does not apply any optimizations and always calls Lucene’s
> {{TermOrdValLeafComparator.copy()}}, which triggers {{LZ4.decompress()}} for
> every document processed by the collapse query. This decompression overhead
> did not exist in Solr 8.
> h1. Solution
> This PR proposes two improvements:
> 1. Load string-sorted doc values lazily for group heads, materializing the
> string only when a competing document appears.
> 2. Avoid loading or materializing string-sorted doc values for documents in
> the same segment during collapse. Use ordinals instead - they’re numeric,
> cheaper to compare, and don’t need decompression.
> The first improvement focuses on scenarios where many collapse groups contain
> only a single document, or where collapse sorting uses multiple fields with a
> string field acting as a tie-breaker.
> The second improvement is expected to deliver major gains in cases where many
> documents originate from the same segment.
> h1. Tests
> Four new tests were added in TestCollapseQParserPlugin:
> - {{testCollapseStringSortLazyLoadingTieDoesNotEvictGroupHead}} - verifies
> that when two documents in the same group have an equal string sort value,
> the first-seen document remains the group head (a tie must not trigger
> eviction). Covers both single-segment (ordinal fast path) and multi-segment
> (slow path) cases.
> - {{testCollapseStringSortOrdinalFastPathMultiClauseTieBreaking}} - verifies
> that when clause-1 of a multi-clause sort ties on ordinal comparison,
> clause-2 correctly decides the winner. Also exercises the remaining-values
> copy loop with a cross-segment competitor.
> -
> {{testCollapseStringSortWithoutDocValuesSkipsLazyLoadingAndOrdinalFastPath}}-
> verifies that sorting on a string field without SORTED DocValues produces
> correct results via the eager field-comparator path, ensuring the lazy
> loading and ordinal fast path are safely bypassed when unavailable.
> - {{testCollapseStringSortOrdinalFastPathDescendingWithMissingValues}} -
> verifies that missing values rank last even under a descending sort, where
> the missing-value sentinel (missingOrd = -1) combined with reverseMul = -1
> must still produce the correct ordering in the ordinal fast path.
> In addition to {{TestCollapseQParserPlugin}}, the {{CollapsingSearch}}
> benchmark was introduced to compare average execution times for collapse
> queries across different sort field combinations. *In the benchmark,
> documents from the same groups are distributed evenly across all segments.*
> The benchmark was executed locally on my machine against Solr branch 10.x and
> Solr branch 10.x-SNAPSHOT, which includes the enhancements.
> <img width="2683" height="1476" alt="benchmark_comparison"
> src="https://github.com/user-attachments/assets/90294b8a-5319-4756-99de-244f78402c8e";
> />
> Conclusions:
> - The {{collapseByDateAndStr}} benchmark shows that Solr 10 SNAPSHOT performs
> significantly better regardless of the number of segments. This is because
> the string field serves only as a tiebreaker in the collapse sort, so in most
> cases comparing dates is sufficient to determine the winner. In addition,
> string doc values are loaded lazily, which avoids eagerly materializing the
> string value when it is not needed. According to the benchmark data, the
> snapshot with the two improvements made {{collapseByDateAndStr}} about 7
> times faster.
> - The {{collapseByStr}} benchmark shows that Solr 10 SNAPSHOT delivers
> significantly better performance only when the number of segments is small,
> especially when most documents from the same group are located in the same
> segment. In the single-segment case, string doc values do not need to be
> materialized to pick a winner, since comparing element ordinals is enough and
> is both safe and efficient. According to the benchmark data, these two
> improvements together made {{collapseByStr}} about 38 times faster for one
> segment. With many segments, however, and with documents from the same groups
> spread evenly across them, the ordinal fast path provides no benefit because
> most comparisons still require string materialization.
> ----
> Transcribed from a PR: https://github.com/apache/solr/pull/4620
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
