jackylee-ch opened a new pull request, #93: URL: https://github.com/apache/paimon-vector-index/pull/93
## Summary `write_ivfrq_index` materialized a full byte copy of each f32 section before writing it. The quantizer centroid section is `nlist * d` floats, bounded only by `MAX_SECTION_ELEMENTS`, so serializing a large index could allocate several GiB transiently on top of the resident centroids. Stream those sections through one reusable 64 MiB write buffer, mirroring #77 for IVF-Flat. ## Testing - New `ivfrq_chunked_writer_preserves_format_and_bounds` asserts byte-identical output and a bounded maximum write. Without the buffer the centroid section goes out as one 32768-byte write and the test fails. - `cargo test --release --workspace` (503 passed), including `storage_format_fixtures` - `cargo fmt --all -- --check`, `cargo +1.98.0 clippy --release --all-targets --workspace -- -D warnings` ## Notes No public API, index option, or file-format change; output is byte-for-byte identical and independent of chunk boundaries. The per-list code and factor buffers built by `block_list` are unchanged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
