airborne12 opened a new pull request, #69790:
URL: https://github.com/apache/doris/pull/69790
### What problem does this PR solve?
Issue Number: None
Related PR: #69051
Problem Summary: score() of a MATCH on an untokenized (keyword) inverted
index is always 0, while SEARCH on the same field scores BM25:
INDEX idx_redirect (redirect) USING INVERTED
-- one row ('BP')
SELECT score() FROM t WHERE redirect MATCH 'BP'; -- 0
SELECT score() FROM t WHERE SEARCH('redirect:BP'); -- 0.2876821
Elasticsearch scores a match, term or match_phrase query on a keyword
field with BM25 in query context: no norms and no frequencies, so
freq = dl = avgdl = 1 and the score is the idf of the whole value
(0.2876821 above, 0.6931471 for N=4, n=2). Only a bool filter scores 0.
SEARCH already gives exactly these values. MATCH on keyword behaved like
a filter, so `body MATCH 'apple' AND tag MATCH 'x'` ranked as if the tag
clause were not there.
Root cause: #69051 made the engine score keyword postings (unit frequency
and norm) but kept MATCH out on purpose: can_score_match required an
analyzed index, and MatchPredicateCollector returned early for indexes
without an analyzer, so no statistics were collected for them.
Fix: a keyword (non-analyzed) index is eligible for MATCH scoring
regardless of support_phrase, and MatchPredicateCollector collects its
whole value as one term (case and spaces kept), the same term the reader
looks up and the same as SearchPredicateCollector collects. Analyzed
indexes keep the existing support_phrase / lowercase-normalizer rule.
Equality and IN stay pure filters. A keyword MATCH_PHRASE_PREFIX scores
the constant 1 of a single-term prefix; MATCH_REGEXP is unchanged.
### Release note
score() of MATCH / MATCH_ALL / MATCH_PHRASE on a keyword (untokenized)
inverted index now returns the BM25 score of the value, the same as
SEARCH and Elasticsearch, instead of 0.
### Check List (For Author)
- Test: Regression test / Unit Test / Manual test
- Regression test: new test_match_keyword_score (V2/V3/SNII, implicit
and explicit parser=none); test_search_score_keyword_filters .out
regenerated: only the keyword MATCH blocks change, now adding idf(x)
and equal to the SEARCH AND blocks; test_omit_norms .out regenerated:
its queries are MATCH_PHRASE_PREFIX on keyword indexes, now 1.0
instead of 0.0. 25 score-related suites in inverted_index_p0, search
and variant_p0 pass
- Unit Test: IndexReaderHelperTest, CollectionStatisticsTest
- Manual test: the reported table on an ASAN build; MATCH equals
SEARCH and Elasticsearch 9.5.5 to 7 digits, also for ARRAY<STRING>,
a column with both a keyword and an analyzed index, and OR/AND mixes
- Behavior changed: Yes. Keyword MATCH contributes its BM25 score to
score() instead of 0.
- Does this need documentation: No
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
