Gabriel39 opened a new pull request, #68375: URL: https://github.com/apache/doris/pull/68375
### What problem does this PR solve? An unfiltered Lance vector search explicitly scoped to every fragment in its snapshot can still scan all indexed row IDs and construct a redundant allow set. This adds work to every query, particularly under concurrency, and the existing profile cannot isolate set construction from input scanning. Integrate the narrowly scoped Lance v11 fix in https://github.com/lance-format/lance/pull/9460 through the existing lance-c patch chain. Pin every Lance crate and the lockfile to commit `f75f3343b5e125c42da1bd7acc8d8217cd5660a6`; the commit is fetchable from the official repository. The upstream PR is pending review. Full-snapshot unfiltered queries omit the row-ID loader; fragment subsets, real predicates, deletion/overlay masks, indexed segment selection, and unindexed fallback retain their semantics. The explicit fragment list is preserved. Expose seven profile metrics: `LancePrefilterLoads`, `LancePrefilterInputRows`, `LancePrefilterInputBatches`, `LancePrefilterRowIds`, `LancePrefilterLoadTime`, `LancePrefilterInputTime`, and `LancePrefilterBuildTime`. Document their scope and overlapping wall-time semantics, and extend indexed multi-vector reader tests to check filtered/non-filtered metrics after close. There is no C ABI or FE protocol change. ### Release note Avoid redundant row-ID prefilter materialization for full-snapshot Lance vector searches and add detailed prefilter profile counters. ### Validation - Lance: 78 prefilter tests and 4 fragment-scope/segment-intersection/FTS tests passed, including execution-summary export, deleted rows, old snapshots, stable row IDs, pinned segments, and unindexed appends. - Lance: workspace rustfmt and `cargo clippy --profile ci --all --tests --benches -- -D warnings` passed. - Doris: syntax compilation of both changed C++ files passed with Lance enabled; clang-format 16 and shell syntax checks passed. - Applied the full eight-patch chain to a fresh lance-c v0.1.9 archive with `--fuzz=0`; Rust 1.91 locked Linux dependency metadata resolved all 21 Lance crates to the pinned commit without updating the lockfile. - The full Doris BE unit-test binary and production concurrency benchmark were not run locally. CI and a benchmark at fixed recall are still needed; no end-to-end latency reduction is claimed. ### Check List (For Author) - [x] Unit Test: extend the existing Lance reader test and add upstream planner/loader tests. - [x] Behavior changed: redundant full-snapshot prefilter work is skipped; query semantics remain unchanged. - [x] Documentation: `docs/lance-prefilter-profile.md` describes the counters and validation method. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
