airborne12 opened a new pull request, #67171:
URL: https://github.com/apache/doris/pull/67171
### What problem does this PR solve?
Problem Summary:
On log-search workloads a query typically carries several pushed-down
conjuncts, e.g. two cheap untokenized MATCH terms plus one expensive `msg
MATCH_PHRASE_PREFIX '...'`:
```sql
SELECT * FROM logs
WHERE _ctime_ >= ... AND _ctime_ <= ...
AND _namespace_ MATCH 'ns' -- untokenized term, ~free
AND _pod_name_ MATCH 'pod-xyz' -- untokenized term, ~free
AND msg MATCH_PHRASE_PREFIX '...' -- whole-segment postings + positions
ORDER BY _ctime_ DESC LIMIT 1000
```
`SegmentIterator` evaluated ALL pushed-down conjuncts against the index
first and only intersected their bitmaps into `_row_bitmap` afterwards, so the
phrase conjunct paid its full whole-segment postings/positions cost even when
an earlier selective conjunct had already emptied the candidate bitmap.
Reproduced on a production log table: one query returning 0 rows burned 11,291
CPU seconds and 124 GiB of local index reads per physical SQL, virtually all of
it inside phrase evaluation (profile: 2,657 segments,
InvertedIndexSearcherSearchExecTime = 99.7% of scan cost, ScanRows = 0).
The empty-bitmap short circuit already exists on the neighbouring paths:
column predicates (`continue_apply` in `_apply_inverted_index`) and compound
AND inside a single expression (`VCompoundPred` COMPOUND_AND early exit). But
top-level AND conjuncts are flattened into separate expr contexts and take
`_apply_index_expr`, which had neither progressive intersection nor a short
circuit, so the most common query shape never benefited.
Fix: intersect each consumed index result into `_row_bitmap` as soon as it
is produced, and stop evaluating further conjuncts once the bitmap is empty. A
skipped conjunct stays pushed down, so the row-level path keeps its exact
semantics over the now-empty candidate set (zero rows read, zero cost).
Consumed conjuncts are erased from `_common_expr_ctxs_push_down` only after the
ANN range-search pass, which iterates the same list. A new profile counter
`InvertedIndexConjunctsShortCircuited` reports how many conjuncts were skipped.
Behavior notes:
- Index results are now intersected before the ANN range-search pass, so ANN
range search executes on the already-narrowed bitmap. AND semantics are
commutative, and a smaller candidate set is what its small-candidate fallback
is designed for.
- Condition cache digest semantics unchanged: a short-circuited (thus still
pushed-down) conjunct keeps the digest cleared, so partial index results are
never cached as full coverage.
### Release note
Inverted index: once earlier pushed-down conjuncts empty the candidate row
bitmap, remaining conjuncts (e.g. expensive MATCH_PHRASE_PREFIX) are no longer
evaluated against the index.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [x] Yes. <!-- Index results are intersected progressively (before the
ANN pass); short-circuited conjuncts fall back to the row-level path over an
empty candidate set. Same query results, strictly less index work. -->
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
