andygrove opened a new issue, #2719: URL: https://github.com/apache/datafusion-ballista/issues/2719
### Is your feature request related to a problem or challenge? `NormalizeInterleaveRule` (`ballista/scheduler/src/state/aqe/optimizer_rule/normalize_interleave.rs`) runs first in the AQE optimizer chain and turns every `InterleaveExec` back into a `UnionExec`. It was added for #2047. AQE rewrites union branches independently, and `InterleaveExec` used to reject a rebuild whose children no longer share one hash or range partitioning. That rejection fired before any rule could repair the parent, so it aborted the replan. DataFusion fixed this upstream in apache/datafusion#24963. `InterleaveExec::replace_children` / `with_new_children` now accept non-interleavable children, report `UnknownPartitioning`, and defer the partitioning check to `InvariantLevel::Executable`. Once Ballista builds against a DataFusion with that change (DataFusion main is pinned in #2718, and it will be in DataFusion 56), the workaround should no longer be needed. ### Describe the solution you'd like - Remove `NormalizeInterleaveRule` and its registration in `AdaptivePlanner::default_optimizers` (`ballista/scheduler/src/state/aqe/planner.rs`). - Keep or port a regression test for the #2047 scenario, for example the TPC-DS queries from that issue under AQE, to confirm that an AQE replan whose union branches diverge still succeeds without the rule. - Check whether plans change. Today the rule always turns an interleave into a union and relies on `EnforceDistribution` to re-form the interleave later, so dropping it may keep interleaves the planner currently loses (or the reverse). ### Describe alternatives you've considered Keep the rule as defensive code. It's cheap, but it rewrites every interleave on every replan to guard against a failure that can no longer happen. ### Additional context Depends on #2718 (or the DataFusion 56 upgrade) landing first. With #2718 applied, the precondition in the rule's own test (rebuilding the interleave with a mismatched child returns an error) already no longer holds, so that check was dropped there. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
