FANNG1 opened a new pull request, #67039:
URL: https://github.com/apache/doris/pull/67039
### What problem does this PR solve?
Issue Number: Part of #66495
Problem Summary:
#66512 gave `vector_search()` end-to-end coverage for all six Lance ANN
algorithms, but only
with **Float32 + L2**. Reviewer feedback there was to merge it and follow up
with the other
element types, which is what this PR does.
The gap is not only a dependency-compatibility one. #66841 made the FE
metric-aware: Doris
plans an indexed split only when the requested metric equals the metric the
index was built
with (`LanceScanNode.metricMatches`), and otherwise plans flat splits rather
than letting
Lance fall back to brute force silently. Until now the **cosine and dot
branches of that
comparison had never run against a real index**, because no non-L2 index
existed anywhere in
the fixture. This PR covers them, in both directions.
### What is changed?
There is no FE or BE production-code change.
#### New indexed tables
One table per remaining metric and per remaining element type, rather than
the full
algorithm x element type x metric cross product, so every Doris code path is
exercised
without multiplying the committed binary fixture:
| Table | Index | Element type | Metric |
|---|---|---|---|
| `vs_ivf_flat_f32_cosine` | IVF_FLAT | Float32 | cosine |
| `vs_ivf_pq_f32_cosine` | IVF_PQ | Float32 | cosine |
| `vs_ivf_pq_f32_dot` | IVF_PQ | Float32 | dot |
| `vs_ivf_flat_f64` | IVF_FLAT | Float64 | l2 |
| `vs_ivf_flat_f16_cosine` | IVF_FLAT | Float16 | cosine |
| `vs_ivf_flat_u8` | IVF_FLAT | UInt8 | hamming |
Float16 uses cosine because building a Float16 L2 index does not complete in
the embedded
Lance version. UInt8 uses hamming with IVF_FLAT because that is the only
combination Lance
accepts for it. Int8 is still absent: Lance has no complete Int8 index build
path, and
searching a nullable Int8 column aborts the BE with the Lance version
`lance-c v0.1.6`
embeds. The compatibility matrix behind these choices is documented in the
companion
doris-website PR.
#### Two new suites
- `test_lance_vector_search_metrics` - the cosine and dot tables. Asserts
through
`lanceSearchIndexSegments` in EXPLAIN that a matching metric plans an
indexed split, and
that a mismatched metric and an omitted metric both do not. Plus the
`nprobes=1`
silent-fallback discriminator per table, and full-probe-equals-flat for
IVF_FLAT.
- `test_lance_vector_search_indexed_element_types` - the Float64, Float16
and UInt8 tables.
`test_lance_vector_search_element_types` deliberately stays on flat search
and covers
query-vector encoding; this is the other half, the same element types read
through an index.
#### Data shapes
The generator is now driven by data profiles rather than one hardcoded
shape, because the
collinear `embedding[j] = (row_id - 1) + j` data the L2 tables use is
degenerate under the
new metrics. Measured with the pinned writer:
- under **dot**, every query returns the same highest-norm rows regardless
of the query;
- under **cosine**, the row directions converge until the top distances
collapse to `0.0` and
the ranking is arbitrary tie-breaking.
So cosine and dot tables use directions from an irrational rotation with an
independently
varying norm, verified tie-free and verified to rank *differently* under l2,
cosine and dot -
that last property is what lets a suite tell a respected metric from an
ignored one. UInt8
uses a thermometer code, which makes Lance's bitwise hamming work out to
exactly `|a - b|`
and reproduces the symmetric ladder the L2 tables have.
#### Writer pin
`pylance` moves from 4.0.1 to 7.0.0, the Lance generation the BE reads
through `lance-c
v0.1.6`. It was previously pinned to the oldest reader so Spark could read
the shared
`__manifest`; that is no longer possible, because on Lance 4 the Float64 and
UInt8 index
builds either panic or produce an index whose every search fails. Spark
compatibility is now
verified by the regression run rather than guaranteed by the pin.
Rebuilding retrains every index, which moved two discriminator rows - the
`ef` discriminator
to row 518 and the UInt8 boundary to row 513. Both are decided by kmeans and
move on any
rebuild, so both self-checks now report the rows that still work when they
fail, instead of
leaving the next person to search by hand.
### Test
- Fixture self-check passes for all 12 tables.
- `external_table_p0/lance`: **12 suites, all passing on a normal golden
comparison**, not
only on generation. The run was done against a MinIO bucket with the lance
prefix cleared
first, so no stale fixture generation could mask a problem. It includes
the Spark and REST
catalog suites that share the Directory Namespace manifest this writer
produced, which is
the check that moving the pin did not break `lance-java 4.0.0` as a reader.
- The goldens reproduce, through Doris, the same rows and distances the
generator measured
directly with pylance: Float64 keeps the exact 0, 16, 64, 144, 256 ladder,
UInt8 gives
hamming 0, 1, 2, 3, 4, and the cosine tables match to float32 display
precision.
Existing goldens moved only where the retrained partitions moved: the
`nprobes=1` boundary
queries now reach the neighbours on the other side of the edge (rows 254,
253, 252, 251 where
they used to be 260, 261, 262, 263). The distance sequence is unchanged,
because the data is
symmetric around the boundary row. No suite assertion changed.
### Release note
None
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
