jayzhan211 commented on issue #26481: URL: https://github.com/apache/datafusion/issues/26481#issuecomment-6661144943
@alamb @2010YOUY01 @zhuqi-lucas Another design I'd like to propose, after alamb's question on #26757: no analyzer at all. Keep every rule in the optimizer, and revisit the rules so they actually satisfy the "Reorderable" invariant we already document on `PhysicalOptimizer` (any order gives a valid plan with the same results; plan quality can still depend on the order). I tried moving every rule to the other side of `EnsureRequirements` on main ([graph and per-rule evidence](https://claude.ai/artifact/HHKNT1vFpviEAtsLEfBGFA)). Most order dependencies only change plans, which is fine under the contract. Only a few rules give wrong results or invalid plans when moved, e.g. `LimitPushdown` before enforcement makes 11 queries return too many rows. So the work would be: - Let the planner pick join modes and add the output-ordering marker, so `JoinSelection` and `OutputRequirements` don't care where they run. - Fix `LimitPushdown`, `PushdownSort`, `CombinePartialFinalAggregate`, `TopKAggregation` and `WindowTopN` so they're correct in any position, without changing plans in the default order. I will try the PoC on this idea -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
