andygrove commented on code in PR #2852:
URL:
https://github.com/apache/datafusion-ballista/pull/2852#discussion_r4601883227
##########
ballista/scheduler/src/cluster/mod.rs:
##########
@@ -604,10 +618,31 @@ fn bind_one(
.map(|bc| bc.max_partitions_per_task())
.filter(|&n| n > 0)
.unwrap_or(usize::MAX);
- let max_partitions = if is_collapse {
+ let split_max = (budget.vcores as usize).min(cap);
+ // The union check rewrites the plan, so it runs only when it can change
+ // the outcome: when the pending partitions don't fit in one slice anyway.
+ let unsplittable_branch =
+ if !is_collapse && running_stage.pending.remaining() > split_max {
+ unsplittable_union_branch(&running_stage.plan, resolver)
Review Comment:
Optional thought. For a wide union stage this runs again on every `bind_one`
call, and each call restricts every branch twice. The answer only depends on
the stage plan and the resolver, so could it be computed once and cached on the
`RunningStage`? Happy for this to be a follow-up if it's fiddly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]