TheR1sing3un opened a new pull request, #9756: URL: https://github.com/apache/paimon/pull/9756
### Purpose The native vector index reader submits multiple positional ranges in one callback, but the Python adapter reads them sequentially. Remote read latency therefore accumulates within a callback. Use a lazy, reusable executor for thread-safe positional reads, preserving range order. The table option `vindex.read.parallelism` bounds active reads per reader across native callbacks, including single-range callbacks. It defaults to 4 for remote index paths and 1 for local paths; seek/read-only streams remain serialized. A value of 1 serializes positional reads across callbacks as well. Wait for submitted reads before propagating an I/O failure, and release the executor on reader close or initialization failure. Document the option and add a reproducible I/O and native-search benchmark. ### Tests - `python -m pytest pypaimon/tests/vindex_input_test.py pypaimon/tests/vindex_vector_index_test.py pypaimon/tests/vector_search_filter_test.py -q`: 88 passed. - Coverage includes concurrent callbacks sharing the read limit, range ordering and cursor preservation, empty/single ranges, seek/read fallback, waiting for outstanding reads after an error, option validation, and cleanup when native construction, initialization, or close fails. - Flake8 with `dev/cfg.ini`, license-header check, and `git diff --check` passed. ### Benchmark macOS arm64, Python 3.9, paimon-vindex 0.4.0. Random seed 42, 16,384 vectors x 64 dimensions, IVF-FLAT nlist=64, nprobe=16, Top-K=10. Baseline uses the original serial positional-read adapter. Timings include reader open, resident initialization, native search, and close. Construction and result assertions are outside the timed region. These are local-file measurements with injected per-read latency, not measurements against a live object store. Twenty repetitions per single-query variant; cells show P50 / P95 milliseconds. | Injected read delay | Baseline | Parallelism 1 | Parallelism 2 | Parallelism 4 | Parallelism 8 | |---|---:|---:|---:|---:|---:| | 0 ms | 0.189 / 0.328 | 0.215 / 0.301 | 0.484 / 0.535 | 0.564 / 0.848 | 0.648 / 0.958 | | 2 ms | 45.338 / 46.523 | 45.330 / 46.522 | 25.590 / 26.492 | 15.853 / 16.245 | 10.769 / 11.400 | | 10 ms | 211.584 / 218.106 | 210.183 / 219.096 | 119.454 / 122.399 | 72.704 / 74.765 | 48.834 / 50.979 | For the 32 x 4 KiB range microbenchmark at 2 ms injected delay, P50 was 78.323 ms for the baseline and 78.512 / 39.954 / 20.364 / 10.403 ms for parallelism 1 / 2 / 4 / 8. This isolates range concurrency from search computation. Local-file thread overhead is why local paths default to 1. With eight queries per batch at 2 ms injected delay, IVF-FLAT P50 fell from 142.437 ms to 41.284 ms at parallelism 4 (10 repetitions). Native IDs/scores and physical read counts/bytes matched across all IVF-FLAT variants: 18 reads and 1,100,414 bytes for one query; 57 reads and 3,660,416 bytes for eight queries. A DiskANN batch check (eight queries, l_search=100, 2 ms injected delay, five repetitions) measured P50 of 99.307 / 194.721 / 59.499 / 38.694 ms for baseline / 1 / 4 / 8. IDs and scores matched. The baseline already reached eight simultaneous reads through native query workers; the new cap applied across callbacks. DiskANN physical read counts varied with shared-cache scheduling (about 73-77 reads per batch), so its I/O totals are not presented as identical. Comparisons use a serial reference for each latency setting because DiskANN also selects its read plan using observed I/O latency. Reproduce with pypaimon[vindex] installed: ```shell python -m pypaimon.benchmark.vindex_io_bench --iterations 20 --output /tmp/vindex-io.json python -m pypaimon.benchmark.vindex_io_bench --iterations 10 --batch-size 8 --latency-ms 0 2 --output /tmp/vindex-io-batch.json python -m pypaimon.benchmark.vindex_io_bench --index-type diskann --iterations 5 --batch-size 8 --latency-ms 2 --parallelism 1 4 8 --output /tmp/vindex-io-diskann.json ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
