bartoszfidrysiak opened a new pull request, #4618: URL: https://github.com/apache/solr/pull/4618
https://issues.apache.org/jira/browse/SOLR-XXXXX (not yet created) # Description Migrating to `Lucene90DocValuesProducer` in Solr 9 revealed a significant performance regression in collapse queries sorted by string fields, due to the extra overhead of LZ4 decompression. Solr 9’s collapse implementation does not apply any optimizations and always calls Lucene’s `TermOrdValLeafComparator.copy()`, which triggers `LZ4.decompress()` for every document processed by the collapse query. This decompression overhead did not exist in Solr 8. ``` SortFieldsCompare.testAndSetGroupValues() / setGroupValues() → TermOrdValLeafComparator.copy() → BaseSortedDocValues.lookupOrd() → TermsDict.seekExact() → TermsDict.decompressBlock() → LZ4.decompress() ← bottleneck ``` # Solution This PR proposes two improvements: 1. Load string-sorted doc values lazily for group heads, materializing the string only when a competing document appears. 2. Avoid loading or materializing string-sorted doc values for documents in the same segment during collapse. Use ordinals instead - they’re numeric, cheaper to compare, and don’t need decompression. # Tests Four new tests were added in TestCollapseQParserPlugin: - `testCollapseStringSortLazyLoadingTieDoesNotEvictGroupHead` - verifies that when two documents in the same group have an equal string sort value, the first-seen document remains the group head (a tie must not trigger eviction). Covers both single-segment (ordinal fast path) and multi-segment (slow path) cases. - `testCollapseStringSortOrdinalFastPathMultiClauseTieBreaking` - verifies that when clause-1 of a multi-clause sort ties on ordinal comparison, clause-2 correctly decides the winner. Also exercises the remaining-values copy loop with a cross-segment competitor. - `testCollapseStringSortWithoutDocValuesSkipsLazyLoadingAndOrdinalFastPath`- verifies that sorting on a string field without SORTED DocValues produces correct results via the eager field-comparator path, ensuring the lazy loading and ordinal fast path are safely bypassed when unavailable. - `testCollapseStringSortOrdinalFastPathDescendingWithMissingValues` - verifies that missing values rank last even under a descending sort, where the missing-value sentinel (missingOrd = -1) combined with reverseMul = -1 must still produce the correct ordering in the ordinal fast path. In addition to `TestCollapseQParserPlugin`, the `CollapsingSearch` benchmark was introduced to compare average execution times for collapse queries across different sort field combinations. The benchmark was executed locally on my machine against Solr 9.10.1 and Solr 9.10.1-SNAPSHOT, which includes the enhancements, and produced the following results: - [solr-9.10.1-collapse-benchmark.log](https://github.com/user-attachments/files/29693835/solr-9.10.1-collapse-benchmark.log) - [solr-9.10.1-SNAPSHOT-collapse-benchmark.log](https://github.com/user-attachments/files/29693837/solr-9.10.1-SNAPSHOT-collapse-benchmark.log) <img width="958" height="603" alt="image" src="https://github.com/user-attachments/assets/75f34afb-9ad0-455a-941b-768ff3f205e9"; /> Conclusions: - The collapseByDateAndStr benchmark shows that Solr 9 SNAPSHOT performs significantly better regardless of the number of segments. This is because the string field serves only as a tiebreaker in the collapse sort, so in most cases comparing dates is sufficient to determine the winner. In addition, string doc values are loaded lazily, which avoids eagerly materializing the string value when it is not needed. According to the benchmark data, the snapshot with the two improvements made collapseByDateAndStr about 9 times faster. - The collapseByStr benchmark shows that Solr 9 SNAPSHOT delivers significantly better performance only when the number of segments is small, especially when most documents from the same group are located in the same segment. In the single-segment case, string doc values do not need to be materialized to pick a winner, since comparing element ordinals is enough and is both safe and efficient. According to the benchmark data, these two improvements together made collapseByStr about 50 times faster for one segment. With many segments, however, and with documents from the same groups spread evenly across them, the ordinal fast path provides no benefit because most comparisons still require string materialization. # Checklist Please review the following and check all that apply: - [ ] I have reviewed the guidelines for [How to Contribute](https://github.com/apache/solr/blob/main/CONTRIBUTING.md) and my code conforms to the standards described there to the best of my ability. - [ ] I have created a Jira issue and added the issue ID to my pull request title. - [ ] I have given Solr maintainers [access](https://help.github.com/en/articles/allowing-changes-to-a-pull-request-branch-created-from-a-fork) to contribute to my PR branch. (optional but recommended, not available for branches on forks living under an organisation) - [ ] I have developed this patch against the `main` branch. - [ ] I have run `./gradlew check`. - [ ] I have added tests for my changes. - [ ] I have added documentation for the [Reference Guide](https://github.com/apache/solr/tree/main/solr/solr-ref-guide) - [ ] I have added a [changelog entry](https://github.com/apache/solr/blob/main/dev-docs/changelog.adoc) for my change -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
