eldenmoon opened a new pull request, #67932:
URL: https://github.com/apache/doris/pull/67932
### What problem does this PR solve?
SEARCH in WHERE is rejected when its input contains an OLAP join. Residual
predicates such as `(MATCH AND EXISTS (...)) OR joined_column = ...` can also
reach execution without an inverted-index evaluation path. Filtering the scan
by MATCH alone would incorrectly remove rows selected by the other OR branch.
This PR binds SEARCH field dependencies before pruning and predicate
movement, and extends the existing scan virtual-column rule to materialize
MATCH/SEARCH booleans used by projections, residual filters, and join
conditions. The original SQL boolean expression and join multiplicity are
preserved. This includes WHERE predicates moved into INNER JOIN conditions by
the optimizer.
It also supports per-field analyzer selection, for example
`search('name@exact:"John Smith" AND title@text:software')`, including
selectors in the `fields` option. Selected index properties use the existing
FE/BE interface. Quoted literal `@` field names remain supported.
Each SEARCH expression still references one table instance; separate SEARCH
expressions can be combined across tables using SQL AND/OR. SEARCH across an
outer join's null-generating side remains conservatively gated. Explicit SEARCH
projections/ON clauses, tuple IN subqueries, analyzer-IN, and multiple
analyzers for the same field within one SEARCH are outside this change.
No BE code, storage format, Thrift, or new plan-node changes are included.
Typed VARIANT TopN uses existing lazy materialization; sorting an untyped
VARIANT value still requires an explicit cast.
### Release note
Support SEARCH predicates in OLAP join queries and per-field analyzer
selection, and evaluate residual MATCH/SEARCH expressions through indexed scan
virtual columns.
### Check List (For Author)
- Test
- [x] Regression test
- [x] Unit Test
- [x] Manual test (details below)
- [ ] No need to test or manual test.
Validation on the original development baseline
`16ab0566e9796d0498e6d2b3221e2a59d5e94ef7` with these changes:
- ASAN BE/FE build and FE Checkstyle passed.
- A 250-test FE matrix passed; a subsequent focused 15-test run passed after
adding INNER JOIN-condition handling (the runs overlap).
- Seven regression suites passed in normal comparison mode:
`test_crm_search_join_document`, `test_crm_search_analyzers`,
`test_crm_search_variant_topn`, `test_search_usage_restrictions`,
`test_search_null_semantics`, `test_search_variant_subcolumn_analyzer`, and
`test_match_projection_virtual_column`.
- Generated expectations cover the CRM query examples,
duplicate/NULL/unmatched join rows, MOW updates, analyzer differences, and
unsupported syntax. Non-equivalent query variants have separate expected
results.
- Typed TopN with and without lazy materialization returned the same 100
ordered rows and payloads; EXPLAIN confirmed MaterializeNode. These are
correctness checks, not performance measurements.
- Disabling the virtual-column rule with MATCH fallback disabled reproduced
`match_any not support execute_match` for the OR/EXISTS control query; enabling
it returned the expected five rows.
For this PR, only the two feature/test commits were cherry-picked onto
master `df36e174b99557a004bc3ad57faa019da4a7d91f`. Range comparison shows
unchanged test changes and only surrounding Analyzer context differences. Tests
have **not** been rerun on this rebased master head; the draft records that
validation boundary explicitly. No C++ files changed, so clang-format is not
applicable.
- Behavior changed:
- [ ] No.
- [x] Yes. SEARCH is accepted in supported OLAP JOIN WHERE contexts;
per-field analyzer selectors are recognized.
- Does this need documentation?
- [ ] No.
- [x] Yes. Follow-up documentation should describe analyzer selectors
and supported SEARCH/JOIN placements.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
