wangzhigang1999 opened a new issue, #86: URL: https://github.com/apache/paimon-vector-index/issues/86
## Motivation Callers that build one vector index file per segment currently need a separate training pass for every writer. The trained centroids, quantizers, and rotations can often be shared when those segments come from the same table or data distribution, but the unified API consumes `VectorIndexTraining` when it creates a writer. For `N` segments, this makes the lifecycle effectively: ```text train -> writer 1 train -> writer 2 ... train -> writer N ``` IVF-PQ already has an internal `from_trained` path, but this capability is not available through the unified API or consistently across the supported index types and language bindings. ## Proposed API Allow a completed training result to create multiple independent writers: ```rust let training = VectorIndexTrainer::train(config, training_vectors, training_count)?; let first = training.create_writer(); let second = training.create_writer(); ``` Each writer should start with the same trained model and an empty vector/ID payload. The existing consuming APIs should remain unchanged for one-shot callers. Equivalent opt-in entry points should be exposed through C, C++, Java/JNI, and Python. ## Expected benefit In a release-mode synthetic build with 128-dimensional vectors, 8,192 training rows, and eight segments of 2,048 vectors each, training once reduced writer preparation time by 84-87% across IVF-Flat, IVF-SQ, IVF-PQ, IVF-RQ, and DiskANN. Including vector insertion and serialization, elapsed time fell by 73-84%. The benefit comes from reducing `N` training passes to one; it does not change the index algorithms themselves. ## Compatibility and scope - No default behavior change: callers opt in to reusable training explicitly. - Existing consuming APIs remain available. - No on-disk format change. - No cross-process training-model serialization or index merging in this issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
