FANNG1 opened a new issue, #67117:
URL: https://github.com/apache/doris/issues/67117
## Problem
Doris may fall back to Flat Search even when a metric-compatible Lance
vector index exists on the queried column.
Lance permits multiple named logical vector indexes on the same vector
column. For example:
```text
vec_l2 column=vector, metric=L2
vec_cosine column=vector, metric=COSINE
```
For a `vector_search()` request with `metric=cosine`, Doris should be able
to select `vec_cosine`. Currently, if `vec_l2` appears first in the index
metadata returned to the FE, Doris selects all segments belonging to `vec_l2`,
checks their metric afterwards, and falls back to fragment-based search when
the metric does not match. It does not continue looking for `vec_cosine`.
The query result remains correct because Lance executes Flat Search with the
requested metric, but the available ANN index is silently missed. Query latency
and resource usage can therefore depend on index metadata ordering.
## Current implementation
`LanceScanNode.selectIndexSegments()` first locks onto the first logical
index covering the vector field:
```java
if (selectedIndexName == null) {
selectedIndexName = segment.getIndexName();
}
if (selectedIndexName.equals(segment.getIndexName())) {
selectedSegments.add(segment);
}
```
Only after that selection does `createIndexSegmentSplits()` call
`metricMatches()`:
```java
List<LanceIndexSegmentInfo> matchingSegments = selectIndexSegments(
metadata.getIndexSegments(), vectorFieldId);
if (matchingSegments.isEmpty() || !metricMatches(vectorSearchParam,
matchingSegments)) {
return Optional.empty();
}
```
Relevant code:
-
`fe/fe-core/src/main/java/org/apache/doris/datasource/lance/source/LanceScanNode.java`
-
`fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceMetadataLoader.java`
The FE already loads the information needed to make a metric-aware decision:
- logical index name
- indexed field IDs
- physical segment UUIDs
- index metric
- fragment coverage
This is not a Lance index-format limitation. Doris can select a logical
index by sending that index's physical segment UUIDs to the BE; no new Lance
`index_name` scanner parameter is required for automatic metric-aware selection.
## Expected behavior
When `use_index=true`, Doris should:
1. Group physical index segments by logical index name.
2. Keep logical indexes covering the requested vector field.
3. Resolve the requested metric using Doris semantics (`L2` when the query
metric is omitted or `DEFAULT`).
4. Select a logical index whose segments use the requested metric.
5. Route only that logical index's segment UUIDs to the BE.
6. Fall back to Flat Search only when no compatible logical index can be
planned safely.
If more than one logical index matches both the field and metric, Doris
should either define a deterministic documented policy or later expose an
optional `index_name` query property. Explicit user selection is a separate
enhancement; it is not required to fix the missed compatible-index case.
Legacy index metadata without a metric or fragment bitmap should continue to
use the existing conservative fallback unless a safe compatibility path is
defined.
## Reproduction
1. Create a Lance dataset with one vector column.
2. Create two named logical indexes on that column, one using L2 and one
using cosine, ensuring the L2 index is returned first by `describeIndices()`.
3. Run Doris `vector_search()` with `metric=cosine` and `use_index=true`.
4. Inspect `EXPLAIN`.
Current behavior:
```text
lanceSearchIndexSegments=0
```
and the query falls back to fragment search, even though the cosine index
exists.
Expected behavior:
```text
lanceSearchIndexSegments>0
```
with only the `vec_cosine` physical segments assigned to indexed splits.
## Acceptance criteria
- [ ] FE unit test: L2 index appears first, cosine index second, and a
cosine query selects the cosine segments.
- [ ] FE unit test: reversing metadata order does not change the selected
metric-compatible logical index.
- [ ] FE unit test: segments from different logical index names are never
mixed in one index plan.
- [ ] FE unit test: no compatible metric still produces the existing Flat
Search fallback.
- [ ] FE unit test: omitted/`DEFAULT` metric follows the documented Doris L2
default.
- [ ] Regression test proves physical indexed execution, using both
`EXPLAIN` segment planning and an `nprobes`/`ef` discriminator rather than only
comparing indexed and flat results.
- [ ] Documentation no longer recommends one index per vector column solely
because Doris cannot find a later metric-compatible index.
## Related
- #66340
- apache/doris#67039 currently uses one vector column per matrix cell to
avoid this selection limitation; it expands coverage but does not change the
production selection logic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
