alamb commented on issue #10456: URL: https://github.com/apache/datafusion/issues/10456#issuecomment-2158908316
> @alamb Is it reasonable to evaluate column in `ConstEvaluator` and collect statistics for `guarantee rewriter` or should we avoid evaluation in logical optimization step and compute it in physical planner? I don't quite follow what you are proposing here. As I I understand the idea on this ticket, the idea is to add a pass that knows how to use [Statistics](https://docs.rs/datafusion/latest/datafusion/common/struct.Statistics.html) to simplify expressions by creating a Simplifier, and pass in the min and max values via [`with_guarantee`](https://docs.rs/datafusion/latest/datafusion/optimizer/simplify_expressions/struct.ExprSimplifier.html#method.with_guarantees) The challenges I see are: 1. statistics not available in the `LogicalPlan` but only in in `ExecutionPlan` via [`ExecutionPlan::with_statistics`](https://docs.rs/datafusion/latest/datafusion/physical_plan/trait.ExecutionPlan.html#method.statistics) 2. The [`ExprSimplifier::simplify`]([simplify](https://docs.rs/datafusion/latest/datafusion/optimizer/simplify_expressions/struct.ExprSimplifier.html#method.simplify)) API is in terms of `Expr`s (not `PhysicalExprs`) One potential thing you could do is use `PruningPredicate` for `FilterExec`s and try to prove inputs can never be true. However, that seems like it may not be particularly effective (as the number of queries where a filter will always be false is likely to be limited in importance) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
