JunRuiLee opened a new pull request, #560:
URL: https://github.com/apache/paimon-rust/pull/560
### Purpose
Part of #514, and the second of three PRs split out of the original combined
change on the primary-key vector `execute_read` path. This PR adds **batch
multi-query search**: N query vectors share one snapshot/manifest plan, segment
preload, and opened readers. Single-query output stays byte-identical (single
== batch-of-one).
> **Stacked PR.** Base is `pk-vec/1-streaming-exact-fallback` (#559). Until
that lands on `main`, the diff here also contains its commit; review the two
top commits in isolation, or wait for the parent to merge. Full stack:
> 1. streaming exact fallback — #559
> 2. **this PR** — batch multi-query
> 3. parallel search — #556 (stacked on this)
### Brief change log
- `feat(vindex)`: the bucket-local ANN search gains a batch entry so N
queries share one pass over each ANN segment and each uncovered file. The ANN
searcher builds the live-row mask once (query-independent), opens one reader
per segment, and drives `visit_batch_vector_search`; the exact searcher scores
all queries in one stream pass. A single query short-circuits to the existing
single-query path so its result is unchanged. The batch ANN scorer dispatches
Lumina vs. vindex readers per segment backend, matching the single-query path.
- `feat(table)`: `BatchVectorSearchBuilder` gains
`with_filter`/`with_projection` and an `execute_read` returning one Arrow
stream per query in input order (empty streams preserve arity; any query error
fails the whole call). The builder factors the query-independent plan
(snapshot, segment preload, residual allow-list, backend, ANN scorer) into a
shared step reused across queries. The single-query
`plan_and_search_pk_candidates` becomes a thin batch-of-one wrapper whose
output is byte-identical. The scored `execute()` stays fail-loud on the
primary-key path, and a data-evolution `execute_read` validates the target
column exists and is a FLOAT vector column (`ARRAY`/`VECTOR`) up front so an
unknown/scalar/non-float column fails loud instead of returning an empty stream.
### Tests
- Single == batch-of-one at every layer; per-query heaps independent under a
shared residual/DV; arity preserved on an empty snapshot; malformed query, zero
limit, and reserved/unknown/scalar/non-float columns fail loud (core + C FFI
coverage).
- `cargo test -p paimon` green; `cargo build -p paimon-datafusion` clean;
`cargo clippy -p paimon --all-targets -D warnings` and `cargo fmt --check`
clean.
### API and Format
No on-disk format change, no new result columns. Single-query output is
byte-identical. New public surface: `BatchVectorSearchBuilder::{with_filter,
with_projection, execute_read}`.
### Documentation
Code comments only; no user-facing docs change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
