hoshinojyunn opened a new pull request, #66979: URL: https://github.com/apache/doris/pull/66979
### What problem does this PR solve? Issue Number: None (Jira: [DORIS-28025](http://39.106.86.136:8090/browse/DORIS-28025)) Related PR: None Problem Summary: A score-sorted inverted-index query on a DUP table can return a deleted document when a delete predicate is present. The issue affects both `SEARCH` and `MATCH_ANY` queries that use `ORDER BY score() ... LIMIT`. Reproduction: ```sql CREATE TABLE test_search_score_topn_delete_predicate ( id INT, title TEXT, INDEX idx_title (title) USING INVERTED PROPERTIES("parser" = "english") ) ENGINE=OLAP DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ( "replication_allocation" = "tag.location.default: 1", "disable_auto_compaction" = "true" ); INSERT INTO test_search_score_topn_delete_predicate VALUES (1, 'alpha alpha alpha alpha alpha alpha'), (2, 'alpha'); SYNC; DELETE FROM test_search_score_topn_delete_predicate WHERE id = 1; SELECT id FROM test_search_score_topn_delete_predicate WHERE SEARCH('title:alpha') ORDER BY score() DESC LIMIT 1; SELECT id FROM test_search_score_topn_delete_predicate WHERE title MATCH_ANY 'alpha' ORDER BY score() DESC LIMIT 1; ``` Before this change, both queries incorrectly return the deleted high-score row: ```text +----+ | id | +----+ | 1 | +----+ ``` The expected result, and the result after this change, is the remaining visible row: ```text +----+ | id | +----+ | 2 | +----+ ``` The scan passed `LIMIT` to the inverted-index score collector before applying delete predicates, so it retained only the deleted document. Separately, score materialization used the same TopN shortcut. Once delete predicates removed that candidate, the visible lower-scoring document had already been discarded and could not be returned. When a segment has delete predicates, this change disables those two score TopN early-truncation paths. The scan collects and materializes all score candidates, evaluates delete predicates, and only then applies the final query limit. The existing score TopN fast path remains unchanged for segments without delete predicates. The regression also verifies that a delete predicate does not affect a later data rowset. ### Release note Fixes incorrect results for score-sorted inverted-index queries with delete predicates. ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [x] Regression test - `./run-regression-test.sh --conf regression-test/conf/regression-conf-custom.groovy --run -g p0 -d search -s test_search_score_topn_delete_predicate` - [x] Manual test (add detailed scripts or steps below) - Rebuilt the `bf-centralize-cloud` cluster from `doris-bf-index:20260820-topn-delete-predicate` and ran the reproduction above for both `SEARCH` and `MATCH_ANY`. - `./build.sh --be -j 192` - Behavior changed: - [ ] No. - [x] Yes. Score TopN is no longer truncated before delete predicates are evaluated; visible lower-scoring rows can satisfy the final limit. - Does this need documentation? - [x] No. - [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
