JunRuiLee opened a new pull request, #556:
URL: https://github.com/apache/paimon-rust/pull/556
### Purpose
Part of #514. Adds three performance capabilities to the primary-key vector
`execute_read` path, mirroring Java
`PrimaryKeyVectorBucketSearch`/`PrimaryKeyBatchVectorRead`, while keeping
single-query output byte-identical:
- **Streaming exact fallback** — bound peak memory to one Arrow batch
instead of the whole vector column.
- **Batch multi-query search** — N query vectors share one snapshot/manifest
plan, segment preload, and opened readers.
- **Intra-search parallelism** — a configurable concurrency limit fans
bucket and exact-file searches out.
### Brief change log
- `perf(table)`: the per-file exact fallback no longer preloads the whole
vector column. It becomes a search closure (`Fn + Send + Sync`) that streams
the column one Arrow batch at a time into per-query bounded top-k heaps, so
peak memory is one batch plus the heaps. The streaming loop lives in the table
layer; the bucket search only calls the closure and merges its bounded results.
Queries are validated before any stream is opened, and the drained row count is
checked against the file metadata in both directions.
- `feat(vindex)`: the bucket-local ANN search gains a batch entry so N
queries share one pass over each ANN segment and each uncovered file. The ANN
searcher builds the live-row mask once (query-independent), opens one reader
per segment, and drives `visit_batch_vector_search`; the exact searcher scores
all queries in one stream pass. A single query short-circuits to the existing
single-query path so its result is unchanged. The batch ANN scorer dispatches
Lumina vs. vindex readers per segment backend, matching the single-query path.
- `feat(table)`: `BatchVectorSearchBuilder` gains
`with_filter`/`with_projection` and an `execute_read` returning one Arrow
stream per query in input order (empty streams preserve arity; any query error
fails the whole call). The builder factors the query-independent plan
(snapshot, segment preload, residual allow-list, backend, ANN scorer) into a
shared step reused across queries. The single-query
`plan_and_search_pk_candidates` becomes a thin batch-of-one wrapper whose
output is byte-identical. The scored `execute()` stays fail-loud on the
primary-key path, and a data-evolution `execute_read` validates the target
column exists and is a FLOAT vector column (`ARRAY<FLOAT>`/`VECTOR<FLOAT>`) up
front so an unknown/scalar/non-float column fails loud instead of returning an
empty stream.
- `perf(table)`: a `global-index.thread-num` option (default 32, rejected
when non-positive) threads a concurrency limit through the search. The
per-bucket loop (orchestrator) and per-exact-file loop (bucket search) fan out
with `buffer_unordered` up to that limit; a limit of 1 takes a plain sequential
loop so visit order stays deterministic. Results are accumulated by fixed query
and split index and merged through the existing order-independent bounded heaps
and global top-k, so output does not depend on completion order. The ANN
segment scan stays sequential (synchronous CPU work).
- `refactor(table)`: the exclusion predicate threaded through the search
(residual filter ∩ deletion vectors) is marked `Send + Sync` so the search
future is `Send` and can run on a multi-threaded runtime.
### Tests
- Streaming search matches the whole-column reference (including a NULL row,
DV/residual exclusion, and heap eviction); row-count truncation/overrun fail
loud.
- Batch: single == batch-of-one at every layer; per-query heaps independent
under a shared residual/DV; arity preserved on an empty snapshot; malformed
query, zero limit, and reserved/unknown/scalar/non-float columns fail loud
(core + C FFI coverage).
- Parallelism: sequential call order asserted at both layers for concurrency
1; parallel (concurrency > 1) with tied/NaN/equal-score candidates completing
out of order produces the deterministic ranking equal to the serial result;
`global-index.thread-num` default/explicit/non-positive.
- `cargo test -p paimon` and `cargo test -p paimon-c` green; `cargo build -p
paimon-datafusion` clean (cross-crate `Send` gate); `cargo clippy --all-targets
-D warnings` and `cargo fmt --check` clean.
### API and Format
No on-disk format change, no new result columns. Single-query output is
byte-identical; the concurrency limit defaults to a value that does not change
results, only scheduling. New public surface:
`BatchVectorSearchBuilder::{with_filter, with_projection, execute_read}` and
the `global-index.thread-num` option.
### Documentation
Code comments only; no user-facing docs change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
