wangzhigang1999 opened a new pull request, #87: URL: https://github.com/apache/paimon-vector-index/pull/87
## Summary Allow one completed `VectorIndexTraining` to create multiple independent writers. Callers building several segment or file indexes from the same data distribution can train once and reuse the trained centroids, quantizers, and rotations instead of repeating training for every writer. The existing consuming APIs remain unchanged for one-shot callers. Reusable writer creation is explicit and opt-in. Closes #86. ## Changes - Add `VectorIndexTraining::create_writer()` for IVF-Flat, IVF-SQ, IVF-PQ, IVF-RQ, and DiskANN. - Initialize each reusable writer with the trained model and empty segment-local IDs, vectors, codes, and graph state. - Expose equivalent non-consuming entry points through C, C++, Java/JNI, and Python while retaining the existing consuming forms. - Document both lifecycle choices without changing the primary one-shot examples. - Cover independent writer payloads with write/open/search round trips and keep binding tests focused on their ownership and ABI boundaries. ## Benchmark Release-mode synthetic benchmark on an Intel Xeon Platinum 8575C host with 32 logical CPUs and 60 GiB RAM. The workload used 128-dimensional vectors, 64 IVF lists, 8,192 training rows, eight segments, and 2,048 vectors per segment. `retrain` trains separately for every segment; `reuse` trains once and creates eight writers. The table reports the median of three alternating process runs. | Index | Prepare: retrain -> reuse | Speedup | Full build: retrain -> reuse | Speedup | | --- | ---: | ---: | ---: | ---: | | IVF-Flat | 112 -> 18 ms | 6.2x | 138 -> 33 ms | 4.2x | | IVF-SQ | 123 -> 20 ms | 6.2x | 140 -> 27 ms | 5.2x | | IVF-PQ | 1,355 -> 173 ms | 7.8x | 1,384 -> 194 ms | 7.1x | | IVF-RQ | 109 -> 16 ms | 6.8x | 132 -> 28 ms | 4.7x | | DiskANN | 1,240 -> 161 ms | 7.7x | 1,485 -> 407 ms | 3.6x | The preparation measurement isolates training and writer creation. The full-build measurement also adds and serializes all segment vectors. Serialized output sizes were unchanged for each index type. This benchmark measures the removal of repeated training work; it does not claim an index-algorithm speedup. ## Testing - [x] `cargo fmt --all -- --check` - [x] `cargo test --workspace --locked` (487 passed, 2 ignored) - [x] `cargo clippy --all-targets --workspace --locked -- -D warnings -A clippy::chunks-exact-to-as-chunks` - [x] C tests (10 passed) - [x] C++ tests (8 passed) - [x] Java Maven tests and JNI native validation - [x] Python 3.12 tests (29 passed) The Clippy allow is limited to the `chunks_exact_to_as_chunks` lint newly enabled by Rust 1.98; strict Clippy currently reports it at 28 unchanged locations on `main`. No call site introduced by this PR requires the allow. ## Compatibility - No default behavior change; reuse requires an explicit new API call. - Existing consuming APIs remain source compatible. - No on-disk format change. - No cross-process training-model serialization or index merging. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
