JingsongLi opened a new pull request, #26: URL: https://github.com/apache/paimon-vector-index/pull/26
## Summary This PR consolidates vector index positional reads behind a single batched `pread` interface and routes IVF reader metadata/list loading through that path. It also batches IVF-PQ probed-list reads and coalesces IVF-HNSW list payload reads to reduce IO round trips under high-latency storage. ## Changes - Replace `SeekRead` seek/read APIs with `pread(&mut [ReadRequest])`, plus an internal `PreadCursor` for sequential header and metadata parsing. - Update core, JNI, and Python stream adapters to implement the new batched read contract. - Batch IVF-PQ single-query and batch-query list reads through `read_inverted_lists` when payload sizes are known. - Keep compatibility fallback for older delta-ID IVF-PQ files that do not store list ID byte lengths in metadata. - Coalesce IVF-HNSW-FLAT and IVF-HNSW-SQ graph list payload reads across single-query and batch-query paths. - Add/update tests covering batched IVF-PQ reads, HNSW coalesced reads, and fallback behavior. ## Testing - [x] `cargo fmt` - [x] `cargo check --workspace` - [x] `cargo check --manifest-path python/Cargo.toml --no-default-features --features auto-initialize` - [x] `cargo test -p paimon-vindex-core` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
