JingsongLi commented on code in PR #9133:
URL: https://github.com/apache/paimon/pull/9133#discussion_r3746906355
##########
paimon-python/pypaimon/common/file_io.py:
##########
@@ -232,12 +237,107 @@ def _read_ranges_coalesced(self, ranges, parallelism,
max_gap, max_span,
coalescible.append((index, path, offset, length))
spans = _coalesce_ranges(coalescible, max_gap, max_span)
+ tasks_by_path = {}
+ for span in spans:
+ tasks_by_path.setdefault(span[0], []).append(("span", span))
+ for singleton in singletons:
+ tasks_by_path.setdefault(singleton[1], []).append(
+ ("one", singleton))
+ task_count = sum(len(path_tasks)
+ for path_tasks in tasks_by_path.values())
+ if task_count == 0:
+ return results
- def _run(task):
+ workers = max(1, min(parallelism, task_count))
+
+ lanes = [[] for _ in range(workers)]
+ lane_loads = [0] * workers
+ path_task_groups = list(tasks_by_path.values())
+ path_capacities = [
+ min(len(path_tasks), _MAX_RANGE_LANES_PER_PATH)
+ for path_tasks in path_task_groups
+ ]
+ path_lane_counts = [1] * len(path_task_groups)
+ remaining_lanes = max(
+ 0,
+ min(workers, sum(path_capacities)) - len(path_task_groups),
Review Comment:
[P2] Do not cap reusable path memberships at the worker count
`remaining_lanes` treats the sum of per-path lane memberships as an
exclusive global budget. However, a physical worker lane can serve multiple
paths sequentially, so these memberships do not need to sum to at most
`workers`. For example, with `parallelism=64`, one path containing 10,000
spans, and 63 paths containing one span each, `len(path_task_groups) ==
workers` makes `remaining_lanes` zero and assigns only one lane to the hot
path. The other 63 workers become idle after their singleton reads while the
hot path stays serial, making the tail close to 16x slower than the per-path
cap permits; the previous proportional allocator did give this path 16 lanes.
Please preserve capped proportional memberships and only water-fill an
under-allocation without shrinking totals above `workers`, or use dynamic
scheduling. A hot-path-plus-63-singletons test should assert 16 hot-path lanes
while global active streams remain at most 64.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
