zhuqi-lucas opened a new issue, #25572: URL: https://github.com/apache/datafusion/issues/25572
### Is your feature request related to a problem or challenge? Follow-up to @alamb's proposal in https://github.com/apache/datafusion/issues/25355#issuecomment-5762888518, filing the independent ticket discussed there. The physical optimizer runs its rule list exactly once, in a hand-ordered sequence. Some passes in that list are not optimizations but enforcers of invariants (`EnsureRequirements`, `OutputRequirements`, the sanity check): they must hold before execution, and any rewrite can invalidate them. Under a once-through model the only way to keep a chain correct is to repeat the enforcement rules by hand after every rewrite that can invalidate them, and the repeats run unconditionally whether or not the rewrite fired. On a production chain with custom rewrites this adds up: 35 physical rule passes of which 28 leave the plan untouched, including six `EnsureRequirements` passes of which four are no-ops, at roughly 80% of physical optimization time. The logical layer does not have this problem because it has both a phase split (`AnalyzerRule` vs `OptimizerRule`) and a convergence loop; the physical layer has neither. ### Describe the solution you'd like Mirror the logical layer, per #25355: 1. **A new trait for enforcement passes** (working name `PhysicalAnalyzerRule`), for rules that uphold invariants rather than improve the plan. The framework runs them at phase boundaries, so downstream chains stop hand-repeating them. 2. **A convergence loop for the remaining optimizer rules**, copying the shape of `Optimizer::optimize`: bounded by `max_passes`, terminated by plan identity. 3. **A `PhysicalPlanSignature`** for that loop. `ExecutionPlan` implements neither `Hash` nor `Eq`, so identity has to be derived; the fingerprint built in #25356 (the rendered plan plus the properties rules consult: partitioning, orderings, equivalences, with statistics deliberately excluded and the boundaries pinned by tests) is a working candidate and can be repurposed wholesale. ### Design questions to settle - **Phase topology.** Is the shape a fixed `optimize-loop → enforce`, or can a chain declare an alternation? Some downstream rules consume the operators enforcement materializes (for example, removing an enforced sort by reorganizing what runs beneath it) rather than the abstract requirement, and today they are deliberately scheduled after an enforcement pass. Either the split sanctions an `enforce → optimize → enforce` alternation, or it establishes the convention that optimizer rules read requirements, never materialized operators. - **Cycle detection, not just "unchanged since last pass".** We have a measured case where an enforcement rule oscillates: two adjacent passes each undo the other's change, so the plan alternates between two forms without stabilizing. The logical loop already handles this by keeping a `HashSet` of every prior pass signature and stopping on first revisit; the physical loop should copy that, not compare only against the previous pass. - **Classification of the built-in rules** into the two traits, and the compatibility story for downstream chains that currently splice enforcement rules in by position. ### Describe alternatives you've considered #25356 (skip a named rule handed a plan it was seen to leave alone) treats the symptom inside the once-through model. Review there converged on the same conclusion as #25355: the phase split is the better home, and the fingerprint work migrates into the convergence check. ### Additional context - https://github.com/apache/datafusion/issues/25355 - https://github.com/apache/datafusion/pull/25356 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
