NoahKusaba opened a new issue, #2445: URL: https://github.com/apache/datafusion-ballista/issues/2445
**Is your feature request related to a problem or challenge?** Multi-partition tasks widen the plan that a single coordinator can see inside one executor. That opens a design question we have no home for: how Ballista's cluster-level AQE should compose with the plan-level adaptivity DataFusion is exploring. This was written up as a "Composition with in-flight DataFusion AQE" subsection in the contributors guide. It was removed in #2438 because it anchored a guide section to two upstream PoCs that have not landed: - apache/datafusion#23026 is still an open draft, last touched 2026-08-24 - apache/datafusion#23167 was closed without merging Filing it here so the thinking survives without the guide claiming it describes current behaviour. **Describe the solution you'd like** The removed text, verbatim: > Two upstream DataFusion PoCs are converging on the same primitives at the > single-plan level: [apache/datafusion#23026](https://github.com/apache/datafusion/pull/23026) > adds `RangeRepartitionExec`, `HaloDropExec`, and a `runtime_partition_extremes` > trait method to parallelize `RANGE`-frame windows inside one plan; > [apache/datafusion#23167](https://github.com/apache/datafusion/pull/23167) adds > `PipelineBreakerBuffer` + `RuntimeOptimizerExec` + a `RuntimeRule` trait so a > plan-root coordinator can observe post-pipeline-breaker runtime stats and > mutate adaptive operators — build-side swaps, split points, skew fixes — in > place, streaming-native, no disk materialization. Multi-partition tasks widens > the plan a single coordinator sees: one `RuntimeOptimizerExec` now observes the > full slice's pipeline-breaker state, so `RangeRepartitionExec`'s halo-aware > routing and any `RuntimeRule`'s adaptive decisions cover an executor's whole > vcore budget instead of one core. Ballista's AQE stage barriers are the > cluster-scale analog of that plan-root coordinator, and the same rule library > lifts unchanged: sketches and row counts collected inside each slice-task get > reported at the shuffle boundary, and the scheduler applies the same rules > cluster-wide. Three levels, one rule library — intra-plan (DataFusion), > intra-executor slice (multi-partition tasks), inter-executor stage boundary > (Ballista AQE). **Describe alternatives you've considered** Keeping it in the contributors guide. That is what #2438 moved away from: the guide should describe code we have, and neither upstream PR is merged, so the section could not be kept accurate. **Additional context** Ballista already carries one piece of this. `PartitionedBoundedWindowAggExec` wraps DataFusion's `BoundedWindowAggExec` to override its `SinglePartition` requirement, and its rustdoc cites apache/datafusion#23026 as the change that would let the wrapper collapse. That link is deliberate and stays: it explains why a workaround exists, rather than describing behaviour we do not have. This issue should be closed if the upstream direction is abandoned, or folded into the AQE epic (#1359) if it is picked up. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
