JunRuiLee opened a new pull request, #645: URL: https://github.com/apache/paimon-rust/pull/645
### Purpose This is PR 1/3 of a vector-search performance series. The vindex search path currently reads the complete index file into memory before opening the index reader. For large indexes, this performs unnecessary I/O and increases peak memory usage even when a query only needs index metadata and a small subset of posting lists. This change introduces a positional-read adapter so vindex can fetch only the ranges requested by `paimon-vindex-core`. ### Brief change log - Add a `SeekRead` implementation backed by Paimon's asynchronous `FileRead`. - Bridge synchronous vindex reads to a dedicated Tokio runtime. - Coalesce nearby positional reads and bound concurrent range requests. - Keep Lumina on its existing buffered read path. - Route regular vindex global-index searches through the range-backed reader. - Preserve existing scalar and batch search behavior. ### Tests - `cargo test -p paimon vindex --lib` - `cargo test -p paimon --test pk_vector_baseline_test` - `cargo test -p paimon --test pk_vector_batch_test` - `cargo test -p paimon --test pk_vector_java_fixture_test` - `cargo fmt --all -- --check` - `git diff --check` ### API and Format No public API or storage-format changes. ### Documentation No user-facing documentation changes are required. This changes the internal vindex I/O strategy only. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
