jayzhan211 commented on issue #26481: URL: https://github.com/apache/datafusion/issues/26481#issuecomment-6416652913
I'd like to try a PoC of what the end state would look like first. With the [current design](https://github.com/apache/datafusion/pull/26757), the analyzer phase is mostly a wrapper around the existing optimizer rules: apart from `OutputRequirements` and `EnsureRequirements`, the rules before enforcement are performance rules that happen to need to run early, so it isn't really an analyzer yet. The current design works, but I'm worried we'd need to change it again later, which could easily turn into another breaking change. I'd like to avoid that. The PoC would make the analyzer contain only correctness rules. Each rule that currently runs before `EnsureRequirements` would either move after enforcement (repaired by `satisfy_requirements`) or be split in two. For example, `join_selection` resolves `PartitionMode::Auto`, which is required for execution, and also swaps inputs based on statistics, which is an optimization. The main question the PoC needs to answer is whether we lose plans by moving rules after enforcement, e.g. `EliminateConstantJoin` losing the pushed-down parquet predicates. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
