zhangstar333 commented on PR #66512: URL: https://github.com/apache/doris/pull/66512#issuecomment-5324556864
> ## Exploratory Doris + Lance compatibility notes > While preparing this PR, I also probed Doris + Lance combinations beyond the committed Float32 + L2 fixtures. This is an integration capability matrix rather than a standalone Lance test: it covers fixture/index creation and Doris queries through FE → BE → lance-c → Lance. The failures found during these probes were traced to Lance-side limitations or bugs; none currently points to a problem in the Doris `vector_search()` implementation. This matrix is retained for follow-up work on #66495 and is not regression coverage added by this PR. > > Environment: indexes generated with pylance 7.0.0 using 1024 × 16 vectors, 4 IVF partitions, and V3 index files, then queried through Doris with lance-c v0.1.6. `yes` means the physical index was created and verified, Doris could read it through lance-c, and the Doris query returned rows through the indexed path. > > Algorithm Float16/L2 Float16/Cosine Float16/Dot Float32/L2 Float32/Cosine Float32/Dot Float64/L2 Float64/Cosine Float64/Dot Int8/L2 UInt8/Hamming > IVF_FLAT hang yes yes yes yes yes yes yes yes no (writer) yes > IVF_PQ hang yes hang yes yes yes yes yes yes no (writer) no > IVF_SQ hang yes yes yes yes yes yes yes yes no (writer) no > IVF_HNSW_FLAT hang yes yes yes yes yes yes yes yes no (writer) yes > IVF_HNSW_SQ hang yes yes yes yes yes yes yes yes no (writer) no > IVF_HNSW_PQ hang yes yes yes yes yes yes yes yes no (writer) no > ### Issues found in the Doris + Lance probes > * **Float16 index-build hangs:** Float16 + L2 can overflow during Lance KMeans centroid accumulation for large, unnormalized values and a low partition count. IVF_PQ + Float16 + Dot can hit a similar overflow in its sub-quantizer training. These are data-dependent Lance index-builder issues, not Doris query-path issues. I am working on a fix and plan to submit it to the Lance community. > * **Nullable Int8 abort:** Searching the existing nullable Int8 fixture aborts the BE because the Lance version used by lance-c v0.1.6 mishandles nulls while converting Int8 vectors. The fix exists in a newer Lance version ([fix(arrow): preserve inner nulls in convert_to_floating_point lance-format/lance#7498](https://github.com/lance-format/lance/pull/7498)), so Doris should add this coverage after upgrading to a lance-c release that includes the fix. > * **Int8 indexing:** Lance currently has no complete Int8 index-build path. This is a Lance writer/core capability limitation, although flat Int8 distance kernels exist. > * **UInt8 indexing:** UInt8 is Hamming-only; the probed Lance version accepts it for IVF_FLAT and IVF_HNSW_FLAT but rejects it for PQ/SQ variants. > * **Invalid Hamming builds:** Building Float32/Float64 indexes with Hamming can reach a Lance KMeans panic instead of returning a validation error. Doris cannot reach this build-time path because Doris only reads and queries Lance indexes. > * **Metric mismatch:** Lance can silently fall back to flat search when the query metric does not match the physical index metric. Therefore compatibility evidence for Cosine or Dot requires a matching physical index and an ANN-execution discriminator; changing only the query metric is not sufficient. > > ### Question for reviewers > This PR currently gives Doris end-to-end coverage for all six ANN algorithms with Float32 + L2. It separately covers Doris query-vector encoding for Float16, Float32, Float64, and UInt8 through flat search. > > Do you think this PR should also add indexed fixtures for non-Float32 element types and Cosine/Dot metrics? These would be Doris + Lance integration tests, but they exercise the same Doris implementation path already covered by Float32 + L2; the additional matrix cells mainly expand dependency compatibility coverage while multiplying the binary fixtures and regression cases significantly. My preference is to keep this PR focused on one representative indexed combination per algorithm and not add the full algorithm × element type × metric matrix, but I would like reviewers' opinions. maybe this pr could merge first, other element types could be laster. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
