airborne12 opened a new pull request, #69698:
URL: https://github.com/apache/doris/pull/69698
### What problem does this PR solve?
Issue Number: None
Related PR: #68758
Problem Summary:
score() over MATCH_PHRASE_PREFIX gave scores that ignored the prefix and
differed between inverted index storage formats.
1. The prefix of a phrase prefix never entered the idf. "apple r" scored on
idf(apple) alone, and a rare expansion such as "rose" ranked like a common
one such as "red". Lucene's multi-phrase query, and so Elasticsearch,
sum the idf of the phrase terms and of every term the prefix expands to.
The statistics collector now registers the prefix; every segment expands
it, counts the document frequency of each expanded term in the tablet
statistics, and records which terms the prefix expanded to. The scorer
adds their idf to the phrase's, so all idfs come from the same tablet
statistics and the score does not depend on how rows are split into
segments. With the reported table, `apple r` now scores 1.4508328, the
value Elasticsearch 9.5 gives; on a 10-row table, `apple r`, `apple ro`
and `apple g` match Elasticsearch to four decimals, over one segment and
over three.
2. A prefix of one token scored 0 on V2/V3 (CLucene did not publish scores
for expansion leaves) and 1 on SNII. It now scores 1 on every format, as
the constant score of Lucene and Elasticsearch prefix queries. At 0,
`score() > 0` dropped every row the prefix matched.
3. No test covered score() over MATCH_PHRASE_PREFIX across storage formats.
Add unit tests for the collector, the segment expansion, the tablet idf
and the constant score, and a regression suite over V2, V3 and SNII.
Two existing tests asserted the old behavior and are updated: the SNII
public phrase prefix scoring test (the idf now counts the expanded term)
and the CLucene one-token phrase prefix test (it asserted no score).
A prefix collected by a SEARCH predicate has no recorded expansion and keeps
scoring the phrase terms alone.
### Release note
score() over MATCH_PHRASE_PREFIX counts the terms the prefix expands to, and
a phrase prefix of one token scores 1 on every inverted index storage format.
### Check List (For Author)
- Test: Unit Test, Regression test
- BE unit tests: 3158 related tests pass; the new idf and segment
expansion tests fail without the change.
- Regression test test_match_phrase_prefix_score on a local cluster.
- Behavior changed: Yes (scores of MATCH_PHRASE_PREFIX change)
- Does this need documentation: No
### Notes for reviewers
- Items 1 and 2 change a design that existing tests asserted on purpose
(`CollectPhrasePrefixExcludesScoringTail`, the SNII phrase prefix scoring
test, `FulltextOneTokenPhrasePrefixPublishesNoScore`). They are updated
here,
with the reason in each test.
- The statistics expand the prefix in every segment with the session's
`inverted_index_max_expansions`, so over several segments the tablet may
hold
more expanded terms than that limit. The union is not cut again, so that
the
idf does not depend on the order of the segments.
- The same bug is fixed separately on branch-4.1 and branch-4.0: those
branches
still have the old phrase query and cannot take this change.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
