airborne12 opened a new pull request, #70008:
URL: https://github.com/apache/doris/pull/70008
Backport of #69790 to branch-4.2.
### What problem does this PR solve?
Issue Number: None
Related PR: #69790, #69051
Problem Summary: score() of a MATCH on an untokenized (keyword) inverted
index is always 0, while SEARCH on the same field scores BM25:
INDEX idx_redirect (redirect) USING INVERTED
-- one row ('BP')
SELECT score() FROM t WHERE redirect MATCH 'BP'; -- 0
SELECT score() FROM t WHERE SEARCH('redirect:BP'); -- 0.2876821
Elasticsearch scores match, term and match_phrase on a keyword field with
BM25 in query context, so the score is the idf of the whole value. #69051
(already on 4.2) made the engine score keyword postings but kept MATCH out.
This makes a keyword index eligible for MATCH scoring and collects its whole
value as one term, the one the reader looks up. It also carries the follow-up
fixes from #69790's review:
- A keyword MATCH_PHRASE drops its ` ~N` slop before collecting.
- A MATCH on a non-string leaf such as `CAST(n AS STRING)` collects nothing.
- A VARIANT path that a segment indexed as BKD, or does not hold, adds no
statistics. This covers the BKD searcher coming from the searcher cache, and
SNII.
- A SNII index kind is looked up without copying the directory.
### Branch adaptations
- `can_score_match` keeps 4.2's analyzed rule (support_phrase only);
branch-4.2 has no lowercase-normalizer clause.
- `test_omit_norms` does not exist on 4.2, so its `.out` change is dropped.
`IndexReaderHelperTest` has no regexp test here, because #69815 is not on 4.2.
- `test_match_keyword_score` inserts its VARIANT rows as JSON strings:
`parse_to_variant` in `VALUES` fails on 4.2 with a max-subcolumns conversion
error. The expected output is master's file, unchanged.
- **#69901 (BM25 statistics per index) is not on 4.2.** A MATCH on a keyword
index and a MATCH on an analyzed index of the same column in one `score()`
query (`name MATCH 'apple' USING ANALYZER english AND name MATCH 'apple' USING
ANALYZER none`) now fails with "Scoring predicates for one field select
different inverted indexes". Master behaved the same before #69901. Before this
change, that query ran with the keyword clause scoring 0. Each clause alone
scores fine. Two analyzed indexes on one column already fail this way on 4.2.
Backporting #69901 lifts this.
### Validation
On branch-4.2 71eecdfc8b + this commit, ASAN build:
- BE UT: 1593 tests in `IndexReaderHelperTest`, `CollectionStatistics*` and
`*Snii*` pass.
- Regression: `test_match_keyword_score` passes against master's `.out` on
V2/V3/SNII. The following also pass: `test_match_phrase_prefix_score`,
`test_bm25_score`, `test_score_filter_matrix`, `test_score_filter_usage`,
`test_search_score_topn_predicates`, `test_multi_analyzer_query`,
`test_match_using_analyzer`, `test_single_column_multi_index` (with debug
points on), `test_search_score_keyword_filters`, `test_search_score_raw_index`.
- Local note: the local thirdparty predates the lance-c upgrade, so
`lance_reader.cpp`'s `lance_scanner_set_scalar_index_auto_selection` call was
stubbed for the local build only. It is not part of this PR.
### Release note
score() of MATCH / MATCH_ALL / MATCH_PHRASE on a keyword (untokenized)
inverted index now returns the BM25 score of the value, the same as SEARCH,
instead of 0.
### Check List (For Author)
- Test: Regression test / Unit Test (see Validation)
- Behavior changed: Yes. Keyword MATCH contributes its BM25 score to score()
instead of 0.
- Does this need documentation: No
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
