guptashivam04 opened a new pull request, #70488:
URL: https://github.com/apache/doris/pull/70488
### Background
`search()` only runs through the inverted index; there is no row-by-row
path, and `vsearch.cpp` turns any index bypass into "SEARCH cannot fall back to
row execution" (added in #66052). For numeric fields (BKD readers),
`InvertedIndexIterator::read_from_index` first estimates the hit count and
returns `INVERTED_INDEX_BYPASS` when it is over `inverted_index_skip_threshold`
percent of the segment (50 by default), so the predicate can be evaluated on
rows instead. That bypass is useful for normal predicates and MATCH, not for
SEARCH.
### What problem does this PR solve?
Issue Number: None
Related PR: #64782 (same fix for the pre-refactor code, closed without
merge), #68758 (moved the scalar leaf into `ScalarLeafCompiler`, bug kept)
PR Type: Bugfix
Problem Summary:
A SEARCH term on a numeric field fails when the value matches more than half
of a segment, for example an account id on a table loaded per account:
```
SELECT count(*) FROM t WHERE search('v.account_id:8069')
ERROR: [E-6001]SEARCH cannot fall back to row execution: [E-6004]hit count:
350255, bkd inverted reached limit 50% , segment num rows:500000
```
`ScalarLeafCompiler::compile` now sets `skip_try` so SEARCH leaves always
read the index. MATCH and plain predicates keep the bypass.
### Details
- **Frequency:** Any SEARCH term on a numeric column or numeric variant
subcolumn whose value covers more than `inverted_index_skip_threshold` percent
of a segment. Common for tenant or account ids. On 4.1 it often shows up only
on the second run, once the exact hit count comes from the query cache.
- **Root cause:** `ScalarLeafCompiler::compile` calls `read_from_index` with
`skip_try = false`, so the BKD hit-ratio check applies to SEARCH even though
SEARCH cannot evaluate rows.
- **Why existing tests missed it:** No SEARCH test runs a numeric term that
covers more than half a segment with the default threshold;
`test_search_inverted_is_null_pushdown` sets `inverted_index_skip_threshold=0`,
which turns the bypass off.
- **Fix:** Set `param.skip_try = true` before `read_from_index` in
`ScalarLeafCompiler::compile`. This is the only `read_from_index` call on the
SEARCH path.
- **Other paths and branches:** branch-4.1 has the same bug in
`FunctionSearch::build_leaf_query`; a separate PR picks this there. Numeric
range clauses (`field:[a TO b]`) are still unsupported in SEARCH; that is not
changed here.
### Testing
- New regression test `search/test_search_bkd_skip_threshold`: a MOW table
where one value covers 70% of the segment, on a variant subcolumn and on a
plain INT column, alone, combined with a text term, in standard mode and with
ORDER BY ... LIMIT, each run 3 times with the query cache on. Fails before the
fix with "SEARCH cannot fall back to row execution", passes after. Ran locally
against a cluster built from this branch, and against the same commit without
the fix.
-
`FunctionSearchTest.TestBuildLeafQueryVariantNestedIntUsesDirectIndexReader`
now checks that the scalar leaf passes `skip_try = true` to the iterator.
Compiled locally with the BE unit-test flags; not run locally (no BE UT build),
left to CI.
- Manual timing on 2M rows (index query cache and SQL cache off, median of
11): a 20% numeric match 10 ms before vs 12 ms after (noise), the 70% match
fails before and takes 16 ms after; text SEARCH and plain filters unchanged.
### Release note
Fix `search()` failing with "SEARCH cannot fall back to row execution" when
a term on a numeric field matches more than `inverted_index_skip_threshold`
percent of a segment.
### Check List (For Author)
- Behavior changed: Yes (see Details). SEARCH no longer follows
`inverted_index_skip_threshold` for numeric fields; queries that failed now
return results.
- Does this need documentation: No
- Pre-submission status: Partially verified: the BE unit test was compiled
but not run locally.
### AI Disclosure
- Tool:
- Generated:
- Human review:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
