airborne12 opened a new pull request, #69699:
URL: https://github.com/apache/doris/pull/69699
### What problem does this PR solve?
Issue Number: None
Related PR: #60331
Problem Summary:
SELECT score() FROM t WHERE c1 MATCH_PHRASE_PREFIX 'apple r' ORDER BY score()
DESC LIMIT 10 fails with "[E3] UnionTermIterator does not support scoring",
while the same filter without score() works.
The last term of a phrase prefix is a UnionTermIterator over the terms the
prefix expands to. PhraseQuery sorts its iterators by document frequency and
scores a row with the norm of the first one. When the terms the prefix
expands
to are rarer than another term of the phrase in a segment, the union comes
first and UnionTermIterator::norm() threw. A union's document frequency is
the
sum of its terms', so whether a query fails depends on the data of each
segment. On a table holding "apple red", "apple green" and "banana yellow",
`apple r` (red in 1 row, apple in 2) and `apple g` fail; on a table where
`r*` matches more rows than apple the same query works. This regressed when
the phrase-level BM25 scoring of #60331 (#61243 in 4.0) started reading the
norm of the first iterator.
A norm belongs to the document, not to a term, so the sub-iterator positioned
on the row, the top of the union's queue, reports it. The union now returns
that norm.
Result on a branch-4.1 cluster built with this change: the reported query
returns 0.4700036 and `apple g` returns a score; on a table where
branch-4.1 (4.1.5) threw for `apple gr`, the query now returns the row that
the same filter without score() returns.
This is not a cherry-pick: master replaced the phrase query engine, and the
same query there scores the prefix on the tablet idf of the terms it expands
to
(#69698).
### Release note
Fix score() over MATCH_PHRASE_PREFIX failing with "UnionTermIterator does not
support scoring" when the prefix matches fewer rows than the other terms.
### Check List (For Author)
- Test: Regression test, Unit Test
- Regression test test_match_phrase_prefix_score on a local branch-4.1
cluster.
- BE unit tests UnionTermIteratorTest.* and PhraseQueryTest.* pass,
including the new NormIsReadFromTheSubIteratorOnTheDocument.
- Behavior changed: Yes (the query no longer fails)
- Does this need documentation: No
### Notes for reviewers
The score this branch returns is still the phrase scored on the idf of its
plain terms (`apple r` returns 0.4700036); it does not count the terms the
prefix expands to. That is left as it was, because it needs the statistics
and
engine changes of master.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
