TheR1sing3un opened a new pull request, #9758: URL: https://github.com/apache/paimon/pull/9758
### Purpose Vector index construction currently materializes the complete shard and training sample before calling the native trainer. Submit the same evenly spaced sample in bounded batches through `VectorIndexTrainer.create`, `add_training_vectors`, and `finish_training`. This removes the complete Python training matrix while preserving sample count/order, null-row handling, relative row IDs and automatic IVF corpus sizing. ### Tests - `python -m pytest pypaimon/tests/vindex_training_test.py pypaimon/tests/global_index_build_test.py -q`: 29 passed. - Native byte-for-byte build comparisons cover IVF-FLAT, IVF-PQ, IVF-SQ, IVF-RQ and DiskANN at sample ratios 1.0 and 0.37 with batches crossing sample boundaries. - Additional tests verify exact sample positions and bounded reads across batch sizes and sampling ratios, plus cleanup when adding training data or finishing training fails. - Flake8, license headers and `git diff --check` passed. ### Benchmark Complete writer builds, including bounded source ingestion, training, adding all production vectors and writing the index. Each run uses a fresh process. macOS arm64, Python 3.9, paimon-vindex 0.4.0; 200,000 vectors x 128 dimensions, IVF-FLAT nlist=64, seed 42. Values are medians of three runs. The sample-matrix ablation uses bounded reads but still allocates the complete training sample, separating the benefit of bounded input reads from native streaming. | Sample ratio | Variant | Peak RSS (MiB) | Finish (s) | Complete build (s) | |---|---|---:|---:|---:| | 1.0 | Original one-shot | 373.328 | 0.211 | 2.523 | | 1.0 | Bounded reads into sample matrix | 372.297 | 0.209 | 2.509 | | 1.0 | Streaming trainer | 280.531 | 0.197 | 2.475 | | 0.1 | Original one-shot | 330.266 | 0.185 | 2.462 | | 0.1 | Bounded reads into sample matrix | 239.125 | 0.178 | 2.452 | | 0.1 | Streaming trainer | 237.078 | 0.180 | 2.441 | All 18 runs produced identical index bytes within each sampling ratio. At ratio 1.0 the complete-build peak falls about 25%; at ratio 0.1 most of the memory reduction comes from bounding the initial read, with a smaller additional benefit from streaming. Build time is approximately unchanged. This bounds Python training buffers, not all native index-construction memory. Reproduce with pypaimon[vindex] installed: ```shell python -m pypaimon.benchmark.vindex_training_bench --output /tmp/training.json ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
