airborne12 opened a new pull request, #67180:
URL: https://github.com/apache/doris/pull/67180
### What problem does this PR solve?
Problem Summary:
The cost of a phrase-family inverted index query (MATCH_PHRASE /
MATCH_PHRASE_PREFIX multi-term path) is proportional to the whole segment's
postings and positions, regardless of how small the surviving candidate set
already is. On a production log table a 9-minute time window left only 22.4%
of each segment's rows after short-key pruning, yet every phrase conjunct
still walked the full segment: per-query profile showed 2,657 segments,
11,913s of InvertedIndexSearcherSearchExecTime (99.7% of scan cost) and
125 GiB of index reads for a query whose final result was 0 rows. A
microbenchmark against the same code path calibrates the cost model to
~0.9µs per co-occurrence candidate, so evaluation cost tracks candidates,
not results.
Fix: expose the scan's current candidate row bitmap to index queries through
IndexQueryContext (the same SegmentIterator -> reader handshake channel the
count-on-index fast path already uses):
- `IndexQueryContext::candidate_rows`: set by SegmentIterator around the
index-apply phase when the candidate bitmap is smaller than
`num_rows * inverted_index_candidate_pushdown_ratio` (new BE config,
default 0.1, <= 0 disables), reset on every exit path via the existing
DEFER. `_row_bitmap` only shrinks during the applies, so restricting to
its current state stays correct for every later conjunct.
- `RoaringDocIdIterator`: a read-only DISI adapter over the candidate
bitmap. PhraseQuery joins it into the leapfrog intersection (its
doc_freq() is the cardinality, so a small candidate naturally becomes the
lead), while matchers keep only real term iterators -- a classic
two-phase iterator: candidates drive the approximation, terms keep the
position semantics. The single-term path is unchanged (no restriction,
same semantics).
- Query cache: a bitmap produced under a non-null candidate is partial and
is never inserted into the query cache. Cache lookups stay enabled -- a
cached full-segment bitmap intersected later is still correct and
cheaper. While a candidate is engaged this also skips caching for
non-phrase fulltext queries (conservative but correct; the restriction
only engages below the ratio threshold where such caching has little
value).
Expected effect on the reproduced workload: with only short-key pruning the
phrase work drops to the in-window fraction of each segment (~4.5x on the
profiled table, more for smaller windows); combined with selective
companion conjuncts the candidate set collapses further and so does the
phrase cost. Results are unchanged -- verified by an equivalence test where
a full-coverage candidate reproduces the unrestricted result.
### Release note
Inverted index: phrase queries now restrict doc-list intersection and
position verification to the scan's surviving candidate rows when the
candidate set is small (BE config inverted_index_candidate_pushdown_ratio,
default 0.1).
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [x] Yes. <!-- Same query results; phrase evaluation cost now tracks
the candidate set. Fulltext query-cache inserts are skipped while a candidate
is engaged. -->
- Does this need documentation?
- [x] No.
- [ ] Yes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
