JunRuiLee opened a new pull request, #535: URL: https://github.com/apache/paimon-rust/pull/535
### Purpose Part of #514. Make primary-key vector search create exact-fallback readers on demand instead of eagerly preloading them, mirroring the on-demand reader creation on the Java read path (`PrimaryKeyVectorBucketSearch`). Previously the read path preloaded an exact-fallback reader for every ANN-uncovered active file before search ran, holding each file's whole vector column in memory even when recall never reached the exact fallback. The preload was a workaround: the reader factory's `create` is `async`, while the bucket-search kernel was synchronous, so the readers had to be `await`ed up front. This change makes the exact-reader factory async (returning a boxed future) through `bucket_search` and the orchestrator, so a reader is built on demand only for a file the fallback actually searches — after the covered / residual / FAST skips have already applied. That removes the builder's eager preload loop and the duplicated "which files need an exact reader" decision, converging it into the one place (the kernel) that already made it. This is a pure IO/memory-peak optimization: output is unchanged (best-first order, Top-K, `_PKEY_VECTOR_SCORE`, projected columns, and the fail-loud guards all hold; the baseline fixture is byte-identical). ### Brief change log - `refactor(table)`: `bucket_search` becomes `async fn`; its exact-reader factory returns a boxed future (`ExactReaderFuture`) instead of a synchronous `Result`. The factory keeps a `Send` bound so the resulting search future stays `Send` for the DataFusion async scan path. - `refactor(table)`: thread the async factory through `PkVectorOrchestrator::search_candidates`. - `perf(table)`: delete the builder's eager exact-reader preload loop and the `should_preload_exact_reader` helper; build readers on demand inside the factory closure (owning all captured data before `.await`). ### Tests - Existing kernel/orchestrator tests preserved unchanged as the behavior regression net; the laziness guard asserts the factory is invoked only for ANN-uncovered files (never for covered or residual-empty files). - `cargo test -p paimon` green (1616 lib + integration, 0 failed), including the PK-vector baseline fixture; `cargo build -p paimon-datafusion` builds (Send path); `cargo clippy -p paimon --lib --tests -- -D warnings` and `cargo fmt --check` clean. ### API and Format - No on-disk format change. No public API change (internal factory seam only). ### Documentation - Doc comments on the affected functions; no user-facing docs change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
