XiaoHongbo-Hope commented on code in PR #9133: URL: https://github.com/apache/paimon/pull/9133#discussion_r3746931946
########## paimon-python/pypaimon/common/file_io.py: ########## @@ -53,6 +55,7 @@ def pread(stream, length: int, offset: int) -> bytes: _COALESCE_GAP = 1 << 20 _COALESCE_SPAN = 8 << 20 _COALESCE_VIEW_MAX_RETAINED_AMPLIFICATION = 2.0 +_MAX_RANGE_LANES_PER_PATH = 16 Review Comment: > Could we document and justify this fixed value? `parallelism` already bounds global concurrency, while this additionally changes single-path behavior: for example, Daft's default `max_concurrency=64` is silently reduced to 16 for one object. The current benchmark uses parallelism 8, so it does not establish why 16 is the right resource/performance trade-off. If there are 64 tasks with parallelism=64 and no per-path cap, the number of stream opens is not reduced, so it cannot solve the QPS issue. That is my concern. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
