airborne12 opened a new pull request, #69801:
URL: https://github.com/apache/doris/pull/69801
### What problem does this PR solve?
Issue Number: None
Related PR: #69698, #69752
Problem Summary:
Backport of #69698 to branch-4.2. score() over MATCH_PHRASE_PREFIX ignored
the prefix and
differed between inverted index storage formats:
1. The idf of a phrase prefix now counts every term the prefix expands to,
as Lucene's
multi-phrase query and Elasticsearch do, and all idfs come from the
tablet statistics. The
statistics expand the prefix in every segment and keep the
`inverted_index_max_expansions`
smallest terms of the tablet that a returned row holds (a session limit
of 0 keeps 50), so a
term's document frequency is counted in every segment that holds it and
merge-on-write hidden
rows do not change the scores after a compaction. A term whose bytes are
not valid UTF-8 is not
scored.
2. A phrase prefix of one token scores 1 on every storage format; on V2/V3
it read 0, so
`score() > 0` dropped every row it matched.
See #69698 for the full description and the review rounds.
**Stacked on #69752**, the branch-4.2 backport of #69438 (BM25 statistics
count only visible
rows of merge-on-write tables), which this change builds on. Until #69752
merges, this PR also
carries its commit; please review only the second commit. After #69752
merges I will rebase.
### Branch adaptations
- No production code adaptation. The automatic pick conflicted only because
#69438 is not on
branch-4.2 yet; on top of #69752 the change applies with one conflict, in
the context of
`collection_statistics_test.cpp`: branch-4.2 already has
`CollectConvertsAnalyzerFailureToStatus` elsewhere in the file, so the
copy that master shows
next to the renamed test is not added again.
### Release note
score() over MATCH_PHRASE_PREFIX counts the terms the prefix expands to, and
a phrase prefix of
one token scores 1 on every inverted index storage format.
### Check List (For Author)
- Test: Regression test, Unit Test
- Validated locally on branch-4.2 at 711db43b00, the commit before
#69625, with the same two
commits on top: the newest branch-4.2 calls a lance-c API that no
thirdparty build on the
machine used here provides. The PR branch itself is on the current
branch-4.2 and is built
by CI.
- test_match_phrase_prefix_score passes with the `.out` of master
unchanged (V2, V3, SNII and
the capped two-segment case).
- test_bm25_score, test_bm25_score_parallel_scan_statistics,
test_bm25_score_range_filter,
test_bm25_score_replica_statistics, test_bm25_score_variant,
test_score_filter_matrix,
test_score_filter_usage and test_search_score_topn_predicates pass.
- 1644 related BE unit tests pass (PhrasePrefixQueryV2Test,
CollectionStatistics*,
PrefixExpansionSetTest, SNII, InvertedIndexReaderTest,
MultiSegmentCollectorTest,
BM25SimilarityTest).
- Behavior changed: Yes (scores of MATCH_PHRASE_PREFIX change)
- Does this need documentation: No
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
