mzabaluev opened a new issue, #27487:
URL: https://github.com/apache/datafusion/issues/27487

   ### Describe the bug
   
   `ExpressionPlacement::MoveTowardsLeafNodes` is documented as a performance 
hint: the expression is cheap and can be pushed down towards the leaves of the 
plan. When the optimizer acts on that hint, it also runs the expression on rows 
that the original plan never evaluated it on:
   
   - `ExtractLeafExpressions` extracts `MoveTowardsLeafNodes` sub-expressions 
from anywhere inside an expression tree, including the branches of `CASE`. In 
the extraction projection, they are evaluated for every input row, not just the 
rows the branch selects.
   - `PushDownLeafProjections` then pushes the extraction projections below 
`Filter` nodes, so the expressions also run on rows the filter removes.
   
   So an expression with this placement must be safe to evaluate on rows that a 
filter or a conditional would exclude: it must not fail on them or have side 
effects. Neither 
[`ExpressionPlacement::MoveTowardsLeafNodes`](https://docs.rs/datafusion-expr-common/latest/datafusion_expr_common/placement/enum.ExpressionPlacement.html)
 nor 
[`ScalarUDFImpl::placement`](https://docs.rs/datafusion/latest/datafusion/logical_expr/trait.ScalarUDFImpl.html#method.placement)
 states this. The built-in functions with this placement (`get_field`, 
`input_file_name`, `file_row_index`) meet the requirement as far as I can tell. 
A user-defined function that returns `MoveTowardsLeafNodes` because it is 
cheap, but fails on some inputs, can make a query fail even though SQL `CASE` 
and `WHERE` semantics say that input is never evaluated.
   
   ### To Reproduce
   
   The plan below shows how `get_field` is moved with the default configuration 
(`datafusion.optimizer.enable_leaf_expression_pushdown = true`):
   
   ```sql
   CREATE TABLE t AS SELECT * FROM (VALUES
     (1,  named_struct('a', 1, 'b', 2)),
     (-1, named_struct('a', 3, 'b', 4))
   ) AS v(x, s);
   
   EXPLAIN SELECT CASE WHEN x > 0 THEN s['a'] WHEN x < -100 THEN 0 END FROM t 
WHERE x <> 5;
   ```
   
   ```
   logical_plan
   01)Projection: CASE WHEN t.x > Int64(0) THEN __datafusion_extracted_1 WHEN 
t.x < Int64(-100) THEN Int64(0) END AS ...
   02)--Filter: t.x != Int64(5)
   03)----Projection: get_field(t.s, Utf8("a")) AS __datafusion_extracted_1, t.x
   04)------TableScan: t projection=[x, s]
   ```
   
   `get_field(t.s, 'a')` came from the first `THEN` branch, and it is now 
evaluated below the filter for every row of `t`.
   
   Replace `s['a']` with a UDF that returns `MoveTowardsLeafNodes` and fails on 
some values, for example a checked conversion that fails when its argument is 
`0`. The query then fails on rows where `x <= 0` or `x = 5`, even though the 
result would never include the UDF's value for those rows.
   
   ### Expected behavior
   
   Either of these would resolve it:
   
   1. **Document the requirement** (preferred). State on 
`ExpressionPlacement::MoveTowardsLeafNodes` and `ScalarUDFImpl::placement` that 
an expression with this placement may be evaluated on rows that enclosing 
filters or conditional expressions would exclude, so it must not fail or have 
side effects on any input row. This matches what the optimizer already relies 
on.
   2. **Stop speculative evaluation**: don't extract leaf expressions from 
conditional sub-expressions (`CASE` branches, the right side of 
short-circuiting `AND`/`OR`), and don't push extraction projections below 
filters when the extracted expression could fail. That would give up much of 
the benefit of the leaf pushdown optimization, which is mainly about evaluating 
`get_field` close to the scan.
   
   ### Additional context
   
   The question came up while reviewing #26589, which precomputes 
`MoveTowardsLeafNodes` expressions over the whole batch before evaluating 
`CASE` branches 
(https://github.com/apache/datafusion/pull/26589#discussion_r4581490634). That 
PR relies on the same assumption at the physical level, so documenting the 
requirement would cover both.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to