alamb commented on code in PR #16861:
URL: https://github.com/apache/datafusion/pull/16861#discussion_r2229107983
##########
datafusion/physical-optimizer/src/filter_pushdown.rs:
##########
@@ -485,21 +497,32 @@ fn push_down_filters(
// currently. `self_filters` are the predicates which are provided by
the current node,
// and tried to be pushed down over the child similarly.
- let num_self_filters = self_filters.len();
- let mut all_predicates = self_filters.clone();
+ // Filter out self_filters that contain volatile expressions and track
indices
Review Comment:
All the book keeping for volatile functions got complicated (though I can't
really see a better way to do it).
Maybe we could encapsulate the book keeping into some structure that would
make it easier to track or something 🤔
##########
datafusion/physical-optimizer/src/filter_pushdown.rs:
##########
@@ -597,3 +631,34 @@ fn push_down_filters(
}
Ok(res)
}
+
+fn allow_pushdown_for_expr(expr: &Arc<dyn PhysicalExpr>) -> bool {
+ let mut allow_pushdown = true;
+ expr.apply(|e| {
+ allow_pushdown = allow_pushdown && allow_pushdown_for_expr_inner(e);
+ if allow_pushdown {
+ Ok(TreeNodeRecursion::Continue)
+ } else {
+ Ok(TreeNodeRecursion::Stop)
+ }
+ })
+ .expect("Infallible traversal of PhysicalExpr tree failed");
+ allow_pushdown
+}
+
+fn allow_pushdown_for_expr_inner(expr: &Arc<dyn PhysicalExpr>) -> bool {
+ // TODO: do downcast matching on specific expressions, or add a method to
the PhysicalExpr trait
+ // to check if the expression is volatile or otherwise unsuitable for
pushdown.
+ if let Some(scalar_function) =
+ expr.as_any()
+ .downcast_ref::<datafusion_physical_expr::ScalarFunctionExpr>()
+ {
+ let name = scalar_function.fun().name();
Review Comment:
I think you can find volatility using
https://docs.rs/datafusion/latest/datafusion/logical_expr/trait.ScalarUDFImpl.html#tymethod.signature
Which then has a volatility field:
https://docs.rs/datafusion/latest/datafusion/logical_expr/struct.Signature.html#structfield.volatility
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
