NoahKusaba commented on code in PR #2852:
URL:
https://github.com/apache/datafusion-ballista/pull/2852#discussion_r4602435269
##########
ballista/scheduler/src/cluster/mod.rs:
##########
@@ -604,10 +618,31 @@ fn bind_one(
.map(|bc| bc.max_partitions_per_task())
.filter(|&n| n > 0)
.unwrap_or(usize::MAX);
- let max_partitions = if is_collapse {
+ let split_max = (budget.vcores as usize).min(cap);
+ // The union check rewrites the plan, so it runs only when it can change
+ // the outcome: when the pending partitions don't fit in one slice anyway.
+ let unsplittable_branch =
+ if !is_collapse && running_stage.pending.remaining() > split_max {
+ unsplittable_union_branch(&running_stage.plan, resolver)
Review Comment:
Done in 5ac2f363e: `RunningStage` keeps the answer from the first `bind_one`
that needs it, so a stage walks its plan once. A stage that fits in one slice
still never computes it. I left a TODO to do the same for
`stage_has_input_collapse`, which has the same pattern on main, and listed it
as a follow-up in the PR body.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]