airborne12 opened a new pull request, #69901:
URL: https://github.com/apache/doris/pull/69901
### What problem does this PR solve?
Issue Number: None
Related PR: #69790
Problem Summary: score() over MATCH clauses that read different inverted
indexes of one column fails:
INDEX idx_english(name) USING INVERTED PROPERTIES("parser" = "english",
...),
INDEX idx_standard(name) USING INVERTED PROPERTIES("parser" =
"standard", ...)
SELECT id, score() FROM t
WHERE name MATCH 'apple' USING ANALYZER english
AND name MATCH 'apple' USING ANALYZER standard
ORDER BY score() DESC LIMIT 10;
-- Scoring predicates for one field select different inverted indexes
BM25 statistics are keyed by field (column unique id plus suffix path),
but each index of a column holds its own terms and lengths: over Apple
and apple, an index that folds case has df(apple) = 2 and one that does
not has df(apple) = 1. One entry cannot serve both, so the collector
refuses the query.
Fix: a MATCH keeps its statistics under the field and the index id.
CollectInfo records the field as indexed for opening the index, since the
map key no longer is one. The readers bind their source to the real field
and pass the statistics name to run_leaf, so the scorers and similarities
need no change: they already look statistics up by the name they are
given. Each clause now scores with the statistics of its own index, and
the combined score is the sum of what each clause scores alone.
SEARCH keeps its own statistics names and its rejection of two indexes on
one field; only MATCH changes. A MATCH and a SEARCH on one index keep
separate entries with the same values.
### Release note
score() over MATCH clauses that use different inverted indexes of one
column, selected with USING ANALYZER, no longer fails.
### Check List (For Author)
- Test: Regression test / Unit Test / Manual test
- Regression test: new test_match_score_multi_index on V2/V3/SNII,
with two indexes that differ in case folding, so df(apple) is 3 in
one and 1 in the other: both clauses score as they do alone, and
identically on the three formats. 26 suites in inverted_index_p0
and search pass, including the multi-analyzer ones
- Unit Test: the collector test that pinned the rejection now pins
both entries; tests that read MATCH statistics use the new key
- Behavior changed: Yes. The query above returns scores instead of an
error.
- Does this need documentation: No
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
