JunRuiLee opened a new pull request, #559: URL: https://github.com/apache/paimon-rust/pull/559
### Purpose Part of #514, and the first of three PRs split out of the original combined change on the primary-key vector `execute_read` path. This PR adds **streaming exact fallback**: it bounds peak memory of the per-file exact rerank to one Arrow batch instead of the whole vector column. Single-query output stays byte-identical. > **Stacked PR (base of the stack).** Follow-ups build on this branch: > 1. **this PR** — streaming exact fallback > 2. batch multi-query — branch `pk-vec/2-batch-multi-query` (stacked on this) > 3. parallel search — #556 (stacked on 2) ### Brief change log - `perf(table)`: the per-file exact fallback no longer preloads the whole vector column. It becomes a search closure (`Fn + Send + Sync`) that streams the column one Arrow batch at a time into per-query bounded top-k heaps, so peak memory is one batch plus the heaps. The streaming loop lives in the table layer; the bucket search only calls the closure and merges its bounded results. Queries are validated before any stream is opened, and the drained row count is checked against the file metadata in both directions. ### Tests - Streaming search matches the whole-column reference (including a NULL row, DV/residual exclusion, and heap eviction); row-count truncation/overrun fail loud. - `cargo test -p paimon` green; `cargo clippy -p paimon --all-targets -D warnings` and `cargo fmt --check` clean. ### API and Format No on-disk format change, no new result columns, no public API change. Single-query output is byte-identical. ### Documentation Code comments only; no user-facing docs change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
