mzabaluev opened a new issue, #27487: URL: https://github.com/apache/datafusion/issues/27487
### Describe the bug `ExpressionPlacement::MoveTowardsLeafNodes` is documented as a performance hint: the expression is cheap and can be pushed down towards the leaves of the plan. When the optimizer acts on that hint, it also runs the expression on rows that the original plan never evaluated it on: - `ExtractLeafExpressions` extracts `MoveTowardsLeafNodes` sub-expressions from anywhere inside an expression tree, including the branches of `CASE`. In the extraction projection, they are evaluated for every input row, not just the rows the branch selects. - `PushDownLeafProjections` then pushes the extraction projections below `Filter` nodes, so the expressions also run on rows the filter removes. So an expression with this placement must be safe to evaluate on rows that a filter or a conditional would exclude: it must not fail on them or have side effects. Neither [`ExpressionPlacement::MoveTowardsLeafNodes`](https://docs.rs/datafusion-expr-common/latest/datafusion_expr_common/placement/enum.ExpressionPlacement.html) nor [`ScalarUDFImpl::placement`](https://docs.rs/datafusion/latest/datafusion/logical_expr/trait.ScalarUDFImpl.html#method.placement) states this. The built-in functions with this placement (`get_field`, `input_file_name`, `file_row_index`) meet the requirement as far as I can tell. A user-defined function that returns `MoveTowardsLeafNodes` because it is cheap, but fails on some inputs, can make a query fail even though SQL `CASE` and `WHERE` semantics say that input is never evaluated. ### To Reproduce The plan below shows how `get_field` is moved with the default configuration (`datafusion.optimizer.enable_leaf_expression_pushdown = true`): ```sql CREATE TABLE t AS SELECT * FROM (VALUES (1, named_struct('a', 1, 'b', 2)), (-1, named_struct('a', 3, 'b', 4)) ) AS v(x, s); EXPLAIN SELECT CASE WHEN x > 0 THEN s['a'] WHEN x < -100 THEN 0 END FROM t WHERE x <> 5; ``` ``` logical_plan 01)Projection: CASE WHEN t.x > Int64(0) THEN __datafusion_extracted_1 WHEN t.x < Int64(-100) THEN Int64(0) END AS ... 02)--Filter: t.x != Int64(5) 03)----Projection: get_field(t.s, Utf8("a")) AS __datafusion_extracted_1, t.x 04)------TableScan: t projection=[x, s] ``` `get_field(t.s, 'a')` came from the first `THEN` branch, and it is now evaluated below the filter for every row of `t`. Replace `s['a']` with a UDF that returns `MoveTowardsLeafNodes` and fails on some values, for example a checked conversion that fails when its argument is `0`. The query then fails on rows where `x <= 0` or `x = 5`, even though the result would never include the UDF's value for those rows. ### Expected behavior Either of these would resolve it: 1. **Document the requirement** (preferred). State on `ExpressionPlacement::MoveTowardsLeafNodes` and `ScalarUDFImpl::placement` that an expression with this placement may be evaluated on rows that enclosing filters or conditional expressions would exclude, so it must not fail or have side effects on any input row. This matches what the optimizer already relies on. 2. **Stop speculative evaluation**: don't extract leaf expressions from conditional sub-expressions (`CASE` branches, the right side of short-circuiting `AND`/`OR`), and don't push extraction projections below filters when the extracted expression could fail. That would give up much of the benefit of the leaf pushdown optimization, which is mainly about evaluating `get_field` close to the scan. ### Additional context The question came up while reviewing #26589, which precomputes `MoveTowardsLeafNodes` expressions over the whole batch before evaluating `CASE` branches (https://github.com/apache/datafusion/pull/26589#discussion_r4581490634). That PR relies on the same assumption at the physical level, so documenting the requirement would cover both. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
