airborne12 opened a new pull request, #66047:
URL: https://github.com/apache/doris/pull/66047
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
BM25 `idf` and `avg_dl` are properties of the whole collection, but
`OlapScanner` built them from
`_tablet_reader_params.rs_splits` — the slice of the read source that *this*
scanner was handed:
```cpp
// be/src/exec/scan/olap_scanner.cpp (before)
if (_tablet_reader_params.score_runtime) {
_tablet_reader_params.collection_statistics =
std::make_shared<CollectionStatistics>();
RETURN_IF_ERROR(_tablet_reader_params.collection_statistics->collect(
_state, _tablet_reader_params.rs_splits, ...)); // <-- only
this scanner's rowsets
}
```
`ParallelScannerBuilder` divides a tablet's `entire_read_source.rs_splits`
into several
`partitial_read_source.rs_splits`, one per scanner. So as soon as parallel
scan produced more than
one scanner for a tablet, each scanner computed `idf` over a *different
subset of rowsets*, and
`score()` depended on how the tablet happened to be split — rows served by a
scanner owning few
rowsets scored differently from identical rows served by a scanner owning
many.
This affects any inverted index format that supports BM25 scoring.
**Fix.** The read source is captured *before* it is sliced, and statistics
are collected once per
tablet over all of its rowsets. `CollectionStatisticsBuildState` performs
the sharing: the first
scanner to arrive builds while the others wait on a condition variable, then
each receives its own
clone. The collected counts live in an immutable block that clones share;
only the lazily memoized
`idf`/`avg_dl` maps are per-scanner, so the read path needs no locking. A
failed or throwing build
is recorded and handed to the waiters rather than retried once per scanner.
The sharing is not incidental — without it this would be a straight
regression, since every scanner
would repeat the full-collection walk that previously covered only its own
slice.
### Release note
Fix `score()` returning different BM25 scores for the same row depending on
how many scanners the
tablet's read source was split into. Collection statistics are now computed
over the entire
collection and shared by all scanners of a tablet.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
`regression-test/suites/inverted_index_p0/test_bm25_score_parallel_scan.groovy`
loads a V3
inverted index table as several rowsets, then compares `ORDER BY
score() LIMIT 20` with
`enable_parallel_scan = false` against the same query forced onto many
scanners
(`parallel_scan_min_rows_per_scanner = 16`,
`parallel_scan_max_scanners_count = 16`).
The two result sets must be identical.
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [x] Yes.
`score()` values change for queries that were previously served by
more than one scanner per
tablet. The new values are the correct collection-wide ones; the old
values varied with the
scanner split. Relative ranking within a single scanner was already
consistent, so this mainly
affects absolute scores and cross-scanner ordering.
- Does this need documentation?
- [x] No.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
