XiaoHongbo-Hope commented on code in PR #9133: URL: https://github.com/apache/paimon/pull/9133#discussion_r3746960197
########## paimon-python/pypaimon/common/file_io.py: ########## @@ -53,6 +55,7 @@ def pread(stream, length: int, offset: int) -> bytes: _COALESCE_GAP = 1 << 20 _COALESCE_SPAN = 8 << 20 _COALESCE_VIEW_MAX_RETAINED_AMPLIFICATION = 2.0 +_MAX_RANGE_LANES_PER_PATH = 16 Review Comment: > Could we document and justify this fixed value? `parallelism` already bounds global concurrency, while this additionally changes single-path behavior: for example, Daft's default `max_concurrency=64` is silently reduced to 16 for one object. The current benchmark uses parallelism 8, so it does not establish why 16 is the right resource/performance trade-off. Got your point. If there are 64 tasks with parallelism=64 and no per-path cap, the number of stream opens is not reduced, so it cannot solve the QPS issue. That is my concern. So I try to add a max values here, In our same-Blob benchmark, it reduced stream opens by 75% compared with 64, with only about 5% end-to-end latency overhead. Lower values increased latency more noticeably, while higher values saved less open/HEAD QPS, so 16 was chosen as the balance point. Will update PR doc with more testing benchmark. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
