XiaoHongbo-Hope opened a new pull request, #613:
URL: https://github.com/apache/paimon-rust/pull/613
## Problem
`vector_search(...)` materializes its results with a plain row-range scan, so
`SELECT * FROM vector_search(...)` returns the matching rows in **physical
file
order** instead of by relevance. This contradicts the documented contract
(`docs/src/sql.md`: *"returns the top-k rows ordered by relevance score"*):
the
first returned row is often **not** the most similar one, which is very
visible at
larger `limit`s.
The vector index already ranks correctly (`execute_scored()` returns
`row_ids`
best-first); the ranking was simply lost when the rows were materialized by a
file-order range scan.
## Fix
In `VectorSearchTableProvider::scan`:
1. `execute_scored()` → global row-ids in best-first relevance order.
2. Read the matching rows **with projection pushdown**, plus the internal
`_ROW_ID`
column.
3. **Gather** the file-order rows back onto the index's existing rank (O(k),
not a
re-sort), and drop `_ROW_ID`. A scored row-id that is not materialized
fails loud
rather than silently shrinking the top-k.
Output schema is unchanged (no score column is exposed). `execute_read()` is
intentionally not reused: it is statically `!Send` because of the
primary-key-vector
path, which the async `TableProvider::scan` cannot hold — so this stays on
the
data-evolution / global-index path, matching this table function's prior
support.
## Test
New regression test `test_vector_search_orders_by_relevance` builds an
ivf-flat
vindex table and asserts the returned order is `[2, 5, 1]` (relevance
order), which
differs from the ascending-id file order `[1, 2, 5]` the old code produced —
covering
both the projected (`SELECT id`) and full (`SELECT *`) paths.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
