Doris-Breakwater commented on issue #69573: URL: https://github.com/apache/doris/issues/69573#issuecomment-6383865497
Breakwater-GitHub-Analysis-Slot: slot_6b733f2d4d73 **Assessment:** The reported mechanism is supported by source inspection of `branch-4.1` at `3f911d030b60ec962a643602ecf103982edd6ca9`. This warrants a Lance integration correctness fix: the distributed merge can compare quantized ANN estimates with exact flat-search distances. This is a code-backed diagnosis, not an independently executed SQL reproduction. The issue is currently open and has no labels. **Verified execution path** - FE selects compatible vector index segments, records indexed/unindexed coverage, then appends uncovered-fragment splits without enabling refinement: [LanceScanPlanner.java, lines 214–255](https://github.com/apache/doris/blob/3f911d030b60ec962a643602ecf103982edd6ca9/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/source/LanceScanPlanner.java#L214). - The TVF sets `refine_factor` only when explicitly supplied. BE forces refinement for multi-vector queries, but single-vector queries only receive the user's option; it independently sets `use_index` from whether the split contains segment UUIDs: [TVF options](https://github.com/apache/doris/blob/3f911d030b60ec962a643602ecf103982edd6ca9/fe/fe-core/src/main/java/org/apache/doris/tablefunction/VectorSearchTableValuedFunction.java#L168), [BE scanner setup](https://github.com/apache/doris/blob/3f911d030b60ec962a643602ecf103982edd6ca9/be/src/format_v2/table/lance_reader.cpp#L1103). - The reported Doris revision pins lance-c `4fafcd64525b274b113208390bbcca86bbbc093b`, whose [Cargo dependencies](https://github.com/lance-format/lance-c/blob/4fafcd64525b274b113208390bbcca86bbbc093b/Cargo.toml#L25) pin Lance `68c12dfd7efe02f90ad2d7f3a239a7eb64884e57`. Its single-vector [C wrapper](https://github.com/lance-format/lance-c/blob/4fafcd64525b274b113208390bbcca86bbbc093b/src/scanner.rs#L565) calls `scanner.refine()` only when the option is present. In the underlying [vector search path](https://github.com/lance-format/lance/blob/68c12dfd7efe02f90ad2d7f3a239a7eb64884e57/rust/lance/src/dataset/scanner.rs#L6327), refinement fetches stored vectors and runs `flat_knn`; otherwise ANN distances survive. For the reported build-index-then-append shape, each indexed split targets fragments covered by its selected segment; its local uncovered-fragment fallback is empty, so that path does not trigger Lance's internal combined-arm rescoring ([coverage handling](https://github.com/l ance-format/lance/blob/68c12dfd7efe02f90ad2d7f3a239a7eb64884e57/rust/lance/src/dataset/scanner.rs#L6521)). - Doris adds an ascending `_distance` TopN over the TVF's candidates ([BindExpression.java](https://github.com/apache/doris/blob/3f911d030b60ec962a643602ecf103982edd6ca9/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindExpression.java#L1875)). Thus cross-split score comparability is Doris's responsibility. The analogous mixed-plan refinement in [lance-spark #860](https://github.com/lance-format/lance-spark/pull/860) supports this approach. **Recommended next step** Have FE mark a finalized vector plan that contains both indexed and uncovered-fragment splits and requires exact rescoring (a selected segment type other than `IVF_FLAT`). Carry that requirement through an optional Lance scan parameter, following the target branch's Thrift field compatibility conventions. In BE, apply refinement to indexed splits before their local result limit, with an effective factor of at least 1 while preserving any larger explicit user value. Keep the existing multi-vector behavior and the distance-bound fallback: this revision deliberately avoids quantized index scans with bounds because candidates can be discarded before rescoring. `refine_factor=1` enables exact rescoring without increasing the ANN candidate budget; it does not make candidate selection exhaustive. Consequently, disagreement with `use_index=false` alone is insufficient proof of this defect. The stronger check is whether each returned indexed row's `_distance` matches its distance computed from the original vector, and whether those scores are correctly merged with uncovered rows. Avoid a universal constant-rescaling fix: quantization error is not generally one fixed multiplier. **Missing runtime evidence / regression acceptance** Please attach a small reproducible dataset or generator, concrete query vector, index-build options (type, metric, quantization settings), pinned dataset version and segment-to-fragment coverage. Include `EXPLAIN` and `(row_id, _distance)` outputs for the default mixed query, the same query with `refine_factor=1`, and `use_index=false`, identifying which rows belong to indexed versus uncovered fragments. The Doris revision is already supplied; report any custom Lance dependency overrides if applicable. No crash log or performance profile is needed to establish this scoring issue. Add a deterministic single-vector quantized fixture with an indexed candidate and an uncovered candidate whose relative order changes when the indexed score is corrected. Assert both mixed coverage and returned distances/order, rather than requiring general ANN equality with exhaustive search. Cover omitted refinement, explicit 1 and a larger factor, plus `IVF_FLAT`, fully indexed, fully flat and multi-vector controls. The cited multi-vector coverage test is protected by refinement; the legacy test's line 84 actually checks a pre-index snapshot, not simultaneous mixed coverage. Until fixed, explicitly adding `refine_factor=1` is a code-supported mitigation for score comparability, with additional vector-read I/O; `use_index=false` provides exhaustive scoring. A coverage policy can be a separate enhancement: STRICT can reject partial coverage, whereas INDEX_ONLY intentionally omits uncovered data and needs that semantic documented. Neither refinement nor this triage establishes full ANN recall or exact equality with a single-process Lance search. No source files were changed and no Doris runtime/regression tests were run during this investigation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
