FANNG1 opened a new issue, #69573: URL: https://github.com/apache/doris/issues/69573
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version branch-4.1 @ 3f911d030b6 ### What's Wrong? When a Lance vector index covers only part of the dataset, `vector_search` plans a *mixed* set of splits: - one split per physical index segment (ANN search, `use_index=true`) - one flat-search split per uncovered fragment (`use_index=false`) `LanceScanPlanner.createVectorIndexSegmentSplits()` builds the index-segment splits and then calls `plan.addUncoveredFragments(visibleFragments.values(), 1)`, and BE derives the mode per split at `be/src/format_v2/table/lance_reader.cpp:1132` (`lance_scanner_set_use_index(scanner, segment_count > 0)`). A Doris global TopN above the scan then merges the union of all splits by `_distance`. The two kinds of split do not report `_distance` on the same scale: - a flat split computes **exact** distances; - an indexed split on a **quantized** index (`IVF_PQ`, `IVF_HNSW_PQ`, `IVF_HNSW_SQ`) computes distances from the quantized codes, i.e. **approximate** values, unless refinement is enabled. This is Lance's documented behaviour: with `refine_factor` unset, Lance does not recompute distances from the original vectors, so `_distance` from a quantized index may be approximate. Doris never enables refinement on its own. `refine_factor` reaches lance-c only when the user passes it (`lance_reader.cpp:1118`, forwarded from `VectorSearchTableValuedFunction.java:181`); the one place Doris forces `refine_factor = 1` is the multi-vector path (`lance_reader.cpp:1105`, guarded by `query.__isset.num_vectors`). So a plain single-vector `vector_search` over a partially indexed table with a quantized index sorts approximate distances against exact ones in one TopN. The ranking is distorted, and the `_distance` column handed back to the user mixes two scales. This is not a Lance defect. Lance defines the approximation and provides `refine_factor` as the remedy, and Lance itself treats "whoever merges the arms owns putting them on one scale" as the rule: see https://github.com/lance-format/lance/pull/9849, which fixed exactly this class of bug inside Lance's LSM vector planner (an exact memtable arm merged with an IVF_HNSW_SQ SSTable arm displaced rows that should have ranked higher) by adding `with_refine_factor` so every approximate arm reports exact distances before the `_distance` merge. Lance cannot fix it for Doris, because Doris is the one that creates the arms. Each BE scanner is a separate process holding exactly one arm; the fact that it is one arm of a larger merge exists only in the FE plan. lance-spark hit the same thing when it moved `VECTOR_SEARCH` onto Spark tasks and resolved it the same way: in a mixed plan, indexed units get `refine_factor = 1` so their distances are re-scored and comparable with the exact flat distances (https://github.com/lance-format/lance-spark/pull/860). ### What You Expected? A single-vector search over a partially indexed table returns the same ordering and the same `_distance` values as the equivalent exact search, within ANN candidate-selection error — it should not be skewed by two distance scales being compared. Concretely: when a vector plan contains both index-segment splits and uncovered-fragment splits, and the selected index is not `IVF_FLAT`, Doris should force refinement on the indexed splits (`refine_factor = max(user_value, 1)`) so every split reports exact distances. Notes for a fix: - `IVF_FLAT` needs nothing; its distances are already exact. FE can tell the difference — the planner already branches on `segment.getIndexType() != IndexType.IVF_FLAT` for the distance-bound check. - The search request is attached per scan node (`TLanceScanParams.external_search_request`), not per split, so this cannot be expressed as a per-split option today. The natural home is BE's existing per-split derivation at `lance_reader.cpp:1132`, where `use_index` is already derived from `segment_count`, driven by one new optional flag from FE saying the plan is mixed (field-id rules in `gensrc/AGENTS.md` apply). - Refinement costs extra I/O, so a complementary knob is worth considering: a coverage mode for vector search mirroring the FTS `coverage_mode=STRICT|INDEX_ONLY`, letting a user refuse a mixed plan outright instead of paying for refinement. - What refinement does **not** fix, and what should be documented rather than chased: the candidate set itself stays approximate (quantized distances can rank a true neighbour out of the top k), and a multi-segment plan probes a larger candidate pool than a single-process Lance scan, so results need not match it exactly. ### How to Reproduce? A Lance table whose quantized vector index covers a subset of fragments — e.g. build an `IVF_PQ` index, then append more rows so the appended fragments are uncovered — and: ```sql SELECT row_id, _distance FROM vector_search( "table" = "<catalog>.<db>.<table>", "column" = "embedding", "query_vector" = "[...]", "top_k" = "10", "metric" = "l2", "use_index" = "true") ORDER BY _distance; ``` `EXPLAIN` shows the mixed plan (`lanceSearchIndexSegments` > 0 together with `lanceSearchUnindexedFragments` > 0). Compare the result against the same query with `use_index=false` (exact), and against the same mixed query with `refine_factor=1` added: the rows from the indexed fragments move, which is the symptom. There is no regression coverage for this shape. Mixed-coverage vector cases do exist — `test_lance_multivector_coverage.groovy:153` (`lanceSearchIndexSegments=2` with `lanceSearchUnindexedFragments=1`) and `test_lance_vector_search_legacy_index.groovy:84` — but every one of them either passes `refine_factor` explicitly or is a multi-vector search, which already forces refinement. A single-vector mixed-coverage case without `refine_factor` is the missing test. ### Anything Else? Related tracking issue: #66340. ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
