jackylee-ch opened a new pull request, #94: URL: https://github.com/apache/paimon-vector-index/pull/94
## Summary `write_index` materialized a full byte copy of each f32 section before writing it. The PQ codebook is `m * ksub * dsub` floats, bounded only by `MAX_SECTION_ELEMENTS`, so serializing a large index could allocate several GiB transiently on top of the resident codebook. Stream the OPQ rotation, quantizer centroids, and codebook through one reusable 64 MiB write buffer, mirroring #77 for IVF-Flat. ## Testing - New `ivfpq_chunked_writer_preserves_format_and_bounds` asserts byte-identical output and a bounded maximum write. Without the buffer the codebook goes out as one 65536-byte write and the test fails. - `cargo test --release --workspace` (503 passed), including `storage_format_fixtures` - `cargo fmt --all -- --check`, `cargo +1.98.0 clippy --release --all-targets --workspace -- -D warnings` ## Notes No public API, index option, or file-format change; output is byte-for-byte identical and independent of chunk boundaries. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
