mzabaluev opened a new issue, #27521:
URL: https://github.com/apache/datafusion/issues/27521

   ### Describe the bug
   
   `get_field` on a `Map` reports `ExpressionPlacement::MoveTowardsLeafNodes` 
placement, but it can fail at execution. Leaf expression pushdown evaluates it 
on rows that a `CASE` branch or a filter would exclude, so a query that should 
succeed returns an error.
   
   This is a built-in instance of the undocumented requirement described in 
#27487: expressions with `MoveTowardsLeafNodes` placement must be safe to 
evaluate on rows that an enclosing filter or conditional would exclude.
   
   ### To Reproduce
   
   ```sql
   create table mt as values (1, map(['a', 'b'], [10, 20])), (2, map(['c'], 
[30]));
   
   SELECT CASE WHEN column1 < 0 THEN column2[7] ELSE 0 END FROM mt WHERE 
column1 <> 5;
   -- DataFusion error: Execution error: The key type Int64 does not match the 
map key type Utf8
   ```
   
   No row satisfies `column1 < 0`, so the lookup should never be evaluated. 
With `SET datafusion.optimizer.enable_leaf_expression_pushdown = false`, the 
query returns `0, 0`. With the default settings, the lookup is taken out of the 
`THEN` branch and evaluated below the filter for every row:
   
   ```
   logical_plan
   01)Projection: CASE WHEN mt.column1 < Int64(0) THEN __datafusion_extracted_1 
ELSE Int64(0) END AS ...
   02)--Filter: mt.column1 != Int64(5)
   03)----Projection: get_field(mt.column2, Int64(7)) AS 
__datafusion_extracted_1, mt.column1
   04)------TableScan: mt projection=[column1, column2]
   ```
   
   Without the `WHERE` clause nothing is extracted, and the same `CASE` returns 
`0, 0`. So whether the query fails depends on an unrelated filter being present.
   
   ### Expected behavior
   
   One of these, preferably the first:
   
   - The planner rejects `column2[7]` with a type error, or coerces the key to 
the map's key type. Then the query either fails consistently at planning or 
succeeds, whichever branches are taken.
   - If a mismatched key is left to execution, the lookup is not evaluated 
speculatively, and the query returns `0, 0` as it does with leaf pushdown 
disabled.
   
   ### Additional context
   
   The error does not depend on the data. `map_lookup` 
(`datafusion/functions/src/utils.rs`) fails when the lookup key's type doesn't 
match the map's key type, and when a nested key has a different structure than 
the map's keys (through `make_comparator`). It fails on any batch it is 
evaluated on, whatever the values are.
   
   The planner could reject this, but it doesn't: 
`GetFieldFunc::return_field_from_args` 
(`datafusion/functions/src/core/getfield.rs`) resolves the value field of a 
`Map` without checking the key argument's type, so `column2[7]` plans 
successfully. The mismatch stays hidden as long as no row reaches the 
expression, and leaf pushdown removes exactly that guard.
   
   Possible fixes:
   
   - Check the key type in `get_field`'s `return_field_from_args`, or coerce 
the key to the map's key type, so a mismatched key is a planning error. As far 
as I can tell, map lookups then cannot fail at execution, and 
`MoveTowardsLeafNodes` would be safe for them.
   - Until then, `get_field` could return `KeepInPlace` when the access path 
reaches into a `Map`. `placement()` only receives the argument placements, not 
their types, so this needs either a type-aware placement API or a check where 
the types are known.
   
   #26589 currently works around this by not precomputing field accesses that 
reach into a `Map` inside `CASE`. That only covers the physical `CASE` 
evaluation, not the logical extraction shown above.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to