Gabriel39 opened a new pull request, #68687: URL: https://github.com/apache/doris/pull/68687
### What problem does this PR solve? Ordinary Lance scans currently retain array label membership predicates in Doris, and scalar index planning skips drivers below OR and NOT. This adds Substrait pushdown for the built-in `array_contains` on a direct `List<Utf8>` column with a non-NULL constant string. Safe boolean combinations can select a LabelList/BTree/Bitmap segment; OR requires a driver on both branches, and NOT requires the complete subtree. For example, two label conditions joined by AND now reach Lance together and can reduce the index candidate set before row materialization. NULL needles, other array types, dynamic operands, UDFs, and `array_contains_all` remain residual. Doris `array_contains_all` tests a contiguous ordered subsequence, so translating it to Lance set containment would change results. Each task still selects one logical index; this change does not implement multi-column index intersection. Fragment coverage, residual evaluation, and limit handling are preserved. Dependency: [lance-format/lance-c#93](https://github.com/lance-format/lance-c/pull/93), pinned at `8d916552fdceb2a13f050f6255fda62e94331688` (currently unmerged). Reapply the existing Foyer payload to that base without changing its implementation. This PR targets `branch-4.1`. Documentation: https://github.com/apache/doris-website/pull/4184 (English and Chinese; draft pending implementation). ### Validation - 60 focused FE JUnit tests passed after compiling the changed converter/planner, the current scan node and metadata sources, and their tests against an existing FE dependency build. Added tests first failed on missing array conversion and OR/NOT driver selection. - FE Checkstyle passed with zero violations. - The pinned lance-c plus the refreshed Foyer patch passed 72 Rust unit tests, 375 C API tests, and 3 native C/C++/static OSS integration tests. - Executed 15 scans using actual Substrait bytes produced by the changed Doris FE through the patched lance-c C API. Checked full/partial/no index coverage, NULL arrays/elements, duplicates, AND/OR/NOT, and two-value IN. The fully indexed label cases searched one segment and reported 8 candidates for one label, 4 for AND, and 12 for OR. - Third-party fresh extraction, cache reuse/refresh, and patch-failure tests passed. The fixture generator and its indexed/unindexed result checks passed; committed fixtures contain only synthetic data and relative paths. - Added `test_lance_array_predicate_pushdown` with result assertions, pushed/residual EXPLAIN assertions, actual segment-search/candidate/fallback Profile assertions, uncovered fragments, and LIMIT coverage. The Groovy suite compiles. Full Doris FE/BE SQL regression execution is pending CI; the C API integration check above is not a substitute for that run. ### Release note Support ordinary Lance array-label membership pushdown and safe boolean scalar index planning. ### Check List (For Author) - Test - [x] Regression test added (SQL execution pending CI) - [x] Unit Test - [x] Manual integration test (FE-produced Substrait through lance-c, described above) - Behavior changed: - [x] Yes. Compatible label predicates are evaluated in Lance; safe OR/NOT expressions can use scalar index segments. - Does this need documentation? - [x] Yes: https://github.com/apache/doris-website/pull/4184 ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Confirm the upstream dependency is ready -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
