Gabriel39 opened a new pull request, #68408: URL: https://github.com/apache/doris/pull/68408
### What problem does this PR solve? Related PR: #68375 (branch-4.1). An explicitly full-snapshot Lance search can materialize a redundant row-ID prefilter. Extract the third-party dependency update from #68375 for master, pinning all Lance dependencies in lance-c's Cargo.toml and Cargo.lock to `f75f3343b5e125c42da1bd7acc8d8217cd5660a6` from https://github.com/lance-format/lance/pull/9460. That upstream PR is still open. The pin stays on Lance v11; it does not upgrade to the newer Lance main branch. The new patch follows master's existing PR #83 patch and has its own marker, so both fresh sources and already-patched source caches receive the update exactly once. The patch payload is identical to #68375. This PR only changes `thirdparty/`: Doris reader profile wiring, documentation, and regression fixture changes remain outside this extraction. The C ABI is unchanged; loader metrics are available upstream but this PR does not expose new Doris profile counters. ### Compatibility The upstream change skips redundant row-ID materialization while retaining explicit fragment scope for fallback and the independent deletion/overlay filtering. Removing the all-row prefilter can activate the existing unfiltered bulk scorer. Its 4-bit IVF_PQ distance-table quantization can change candidates and reduce recall at the same refinement budget. #68375 documents a fixture-level recall change and carries a separate regression workaround, which is not included here. This PR does not fix PQ scoring or claim unchanged recall. Recall-sensitive deployments should benchmark the affected query shape; increasing refinement may help but does not guarantee identical results. ### Release note Update the Lance dependency used by lance-c to avoid redundant full-snapshot row-ID prefilters and provide upstream prefilter loader metrics. The existing 4-bit IVF_PQ bulk scoring path may produce different ANN candidates as described above. ### Validation - Verified the lance-c v0.1.9 source archive against the checksum in `thirdparty/vars.sh`. - Executed the actual Lance patch block on fresh sources and on sources prepared by master's previous patch block, with `--fuzz=0`. - Repeated both paths to verify idempotence and compared the resulting lockfiles. - `cargo +1.91.0 metadata --locked --offline --filter-platform x86_64-unknown-linux-gnu`: all 21 Lance-source packages resolve to the intended revision without a lockfile update. - `bash -n thirdparty/download-thirdparty.sh` and `git diff --cached --check` passed. - Full third-party compilation, Doris BE tests, and external regression tests were not run for this extraction; CI validation remains required. ### Check List (For Author) - Test: dependency patch application, cached-source upgrade, rerun idempotence, locked dependency resolution, and shell syntax checks. - Behavior changed: Yes; the dependency removes redundant prefilter work and can select the existing 4-bit PQ bulk scoring path. - Does this need documentation: Compatibility and dependency-only scope are documented above; Doris profile documentation remains with #68375. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
