airborne12 opened a new pull request, #69825:
URL: https://github.com/apache/doris/pull/69825
### What problem does this PR solve?
Issue Number: None
Related PR: #69815, #69269
Problem Summary: score() of pattern queries disagrees with Elasticsearch
and with itself on branch-4.1:
-- 'Inverted indexes are suitable for full text search',
-- 'The parser splits text into multiple terms', ...
SELECT score() FROM document WHERE SEARCH('content:/.*ext/'); -- BM25
sum
SELECT score() FROM document WHERE SEARCH('content:*ext'); -- BM25
sum
SELECT score() FROM document WHERE content MATCH_REGEXP 'ext'; -- 0
Elasticsearch scores a query_string /regexp/, wildcard or prefix, and a
regexp, wildcard or prefix query, with a constant 1 per matching row
(default rewrite constant_score_blended); measured on 9.5.5 with the
rows above: 1.0 for both matching rows. Only an explicit
"rewrite": "scoring_boolean" sums the BM25 of the expanded terms. Next
to a scored clause in a bool must, a regexp adds exactly 1.
#69269 (the branch-4.1 pick of #69052) made SEARCH regexp and wildcard
sum the BM25 of every expanded term, i.e. scoring_boolean always on,
with idf and avgdl taken from the current segment rather than the tablet
statistics MATCH uses, so identical rows in two segments scored
differently. MATCH_REGEXP never requested a scoring context, so it
scored 0 like a filter.
Fix, the same as #69815 on master:
1. Revert #69269: RegexpWeight scores every pattern with the constant
scorer again; its suite that pinned the BM25 sums is removed.
2. MATCH_REGEXP requests scoring on every index, and the legacy
RegexpQuery records 1 for every matching row when score() is
requested. The MATCH statistics collector skips it, since a constant
reads no statistics and the pattern is not text to analyze.
A MATCH_REGEXP on a keyword (untokenized) index also scores 1 once the
keyword reader requests scoring (#69791); until then it keeps 0.
### Release note
score() of SEARCH regexp and wildcard clauses is a constant 1 per
matching row again, like prefix clauses and Elasticsearch's default.
score() of MATCH_REGEXP on a tokenized index is 1 instead of 0.
### Check List (For Author)
- Test: Regression test / Unit Test / Manual test
- Regression test: new search/test_search_score_pattern_constant
(V2/V3, two segments); all 18 blocks equal the master PR's V2/V3
blocks; 16 existing score suites in inverted_index_p0 and search
pass unchanged
- Unit Test: 130 tests (IndexReaderHelperTest, CollectionStatistics*,
RegexpQueryTest and the query_v2 regexp/wildcard tests restored by
the revert)
- Manual test: Elasticsearch 9.5.5 on the same rows; a two-segment
table now scores 1 for every pattern hit on a branch-4.1 ASAN build
- Behavior changed: Yes. SEARCH regexp/wildcard scores and MATCH_REGEXP
scores change as described in the release note.
- Does this need documentation: No
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
