TheR1sing3un opened a new pull request, #9798: URL: https://github.com/apache/paimon/pull/9798
### Purpose PyPaimon's generic global-index builder currently builds vector and full-text shards one at a time. On multi-shard builds this leaves storage bandwidth and native index capacity idle. Add `global-index.build.parallelism` (default `1`) and use a bounded thread pool when the value is greater than one. Each worker keeps the existing streaming Arrow reader and one writer per shard, manifest messages are returned in shard-plan order, and a failed build closes active resources and removes every uncommitted index file. The documentation warns that each shard may also start native worker threads. ### Benchmark A temporary local benchmark exercised the production shard scheduler and streaming batch path with 4,000 approximately 2 KiB text rows per shard. It injected 20 ms of read latency plus 40 ms of native-build/upload latency per shard to isolate the scheduling change. Results are medians of three runs on macOS arm64; RSS is peak resident memory. | Shards | Parallelism | Time (s) | Speedup | Peak RSS (MiB) | |---:|---:|---:|---:|---:| | 4 | 1 | 0.448 | 1.00x | 70.2 | | 4 | 2 | 0.287 | 1.56x | 99.6 | | 4 | 4 | 0.207 | 2.16x | 150.3 | | 8 | 1 | 0.763 | 1.00x | 70.2 | | 8 | 2 | 0.456 | 1.68x | 99.5 | | 8 | 4 | 0.294 | 2.59x | 157.7 | | 8 | 8 | 0.238 | 3.21x | 269.3 | | 16 | 1 | 1.421 | 1.00x | 70.2 | | 16 | 2 | 0.783 | 1.82x | 99.7 | | 16 | 4 | 0.470 | 3.02x | 157.8 | | 16 | 8 | 0.339 | 4.20x | 274.4 | The ablation shows the throughput/memory tradeoff behind the conservative default. The benchmark uses simulated native and I/O latency, so end-to-end gains will depend on the index implementation, storage, shard size, and native thread settings. The temporary benchmark harness is not included in this change. ### Tests - `python -m pytest -q pypaimon/tests/global_index_build_test.py` (36 passed) - `python -m flake8 --config=dev/cfg.ini pypaimon/common/options/core_options.py pypaimon/globalindex/create_global_index.py pypaimon/tests/global_index_build_test.py` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
