FANNG1 commented on PR #66512:
URL: https://github.com/apache/doris/pull/66512#issuecomment-5314024809
## Exploratory compatibility matrix (not committed regression coverage)
The following matrix records exploratory probes performed while preparing
this PR. It is
kept here for follow-up work on #66495, but it should **not** be read as
Doris regression
coverage provided by this PR.
Environment: pylance 7.0.0, 1024 x 16 vectors, 4 IVF partitions, V3 index
files. `yes`
means the index was created, the plan contained
`ANNSubIndex`/`ANNIvfPartition`, and a
search returned rows.
| algorithm | f16/L2 | f16/cosine | f16/dot | f32/L2 | f32/cosine | f32/dot
| f64/L2 | f64/cosine | f64/dot | int8/L2 | uint8/hamming |
|---|---|---|---|---|---|---|---|---|---|---|---|
| IVF_FLAT | hang | yes | yes | yes | yes | yes | yes | yes | yes | no
(writer) | yes |
| IVF_PQ | hang | yes | hang | yes | yes | yes | yes | yes | yes | no
(writer) | no |
| IVF_SQ | hang | yes | yes | yes | yes | yes | yes | yes | yes | no
(writer) | no |
| IVF_HNSW_FLAT | hang | yes | yes | yes | yes | yes | yes | yes | yes | no
(writer) | yes |
| IVF_HNSW_SQ | hang | yes | yes | yes | yes | yes | yes | yes | yes | no
(writer) | no |
| IVF_HNSW_PQ | hang | yes | yes | yes | yes | yes | yes | yes | yes | no
(writer) | no |
### Findings and boundaries
1. **Float16 + L2 can hang during index creation, but this is
data-dependent.**
With the probed data magnitude (~10^3 per component) and 4 partitions,
Float16
accumulation overflows during KMeans centroid calculation. Distances
become `inf`, all
clusters can remain empty, and cluster splitting does not terminate. This
is not an
unconditional Float16/L2 limitation:
- 4 partitions with the original magnitude: hangs for all six algorithms;
- 64 partitions with the original magnitude: builds in about 0.7-0.8
seconds;
- 4 partitions with values scaled by 0.1: builds in about 0.7-0.8 seconds.
Normalized embeddings are therefore not represented by the failing case.
Doris does
not build Lance indexes, so this constrains fixture generation rather
than the BE query
path.
2. **IVF_PQ + Float16 + Dot has a separate KMeans overflow.**
PQ trains its sub-quantizer on raw vectors for Dot, so the same large
values reach a
second KMeans stage. Scaling values by 0.1 avoids the hang; increasing
the IVF partition
count does not, because the PQ sub-centroid count is independent of that
setting.
3. **Int8 is searchable in Lance core but is not indexable.**
pylance rejects Int8 vector index creation, and Lance core has no
complete Int8 storage
or quantizer path for these indexes. Flat Int8 distance kernels exist,
but the Python
binding coerces integer query vectors to Float32, making that path
inaccessible through
pylance without bypassing the coercion.
Doris also cannot safely add the nullable Int8 flat-search regression
yet: with the
Lance version used by lance-c v0.1.6 it aborts the BE while converting
null Int8 vectors.
The upstream fix is lance-format/lance#7498 (`15bbd4a85`), first released
after the Lance
version currently pinned by lance-c.
4. **UInt8 is Hamming-only and only the FLAT storage variants accept it.**
IVF_FLAT and IVF_HNSW_FLAT build with UInt8/Hamming. PQ and SQ reject
UInt8. UInt8 with
L2, Cosine, or Dot is rejected by the query validation path.
5. **Hamming with Float32/Float64 is rejected too late during index
creation.**
The probed Lance version can reach a KMeans panic when building a
floating-point index
with Hamming instead of returning a normal validation error. Querying an
existing
floating-point index with Hamming does return a clean unsupported-metric
error, so Doris
cannot reach this build-time panic through `vector_search()`.
6. **A metric mismatch silently falls back to flat search.**
Querying an L2 index with Cosine or Dot can return correct-looking rows
through brute
force instead of proving that the requested metric works with the index.
Therefore each
metric needs its own physical index plus a discriminator such as low
`nprobes`/`ef`;
changing only the query metric is not compatibility evidence.
7. **Some follow-up fixtures require a newer writer.**
The current pylance 4.0.1 fixture writer cannot reliably produce several
Float64 and
UInt8/Hamming graph-index cells that work under pylance 7.0.0. Moving the
writer pin must
also re-verify the Spark/lance-java reader and the shared Directory
Namespace manifest.
### What this PR actually commits
The committed Doris end-to-end index matrix remains all six algorithms with
**Float32 +
L2 only**. Float16/Float32/Float64 and UInt8 in the element-type suite are
flat-search
encoding tests, not indexed compatibility tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
