mzabaluev opened a new issue, #27534: URL: https://github.com/apache/datafusion/issues/27534
### Is your feature request related to a problem or challenge? `ExpressionPlacement::MoveTowardsLeafNodes` means "a cheap expression that can be pushed to leaf nodes". The docs justify the hint with struct field access, which is "just an Arc clone of the nested array". `GetFieldFunc::placement` returns `MoveTowardsLeafNodes` whenever the base argument is a column (or already leaf-placeable) and every key is a literal. It does not distinguish a struct from a map, because `ScalarUDFImpl::placement` only receives the placements of its arguments: ```rust fn placement(&self, args: &[ExpressionPlacement]) -> ExpressionPlacement ``` A placement says whether an argument is a `Column`, a `Literal`, and so on, but not what type it has. So `s['a']` on a struct and `m['a']` on a map both come out as `MoveTowardsLeafNodes`. A map lookup is not cheap. `map_lookup` scans the entries of every row for a matching key, so its cost grows with the total number of entries in the batch. A struct field access does no per-row work. Because the placement is wrong for maps, the optimizer treats a map lookup like a struct field access: - `ExtractLeafExpressions` / `PushDownLeafProjections` move it out of conditional branches and below filters, so it is evaluated for rows that would not otherwise reach it. For example, `CASE WHEN rare_condition THEN m['k'] END` scans the map entries of every row instead of only the selected few. - `common_subexpr_eliminate`, `push_down_filter` and the physical `projection_pushdown` all use the same placement and make the same assumption. The cost of that extra work grows with map size and with how selective the branch or filter is. The wasted work is largest when few rows need the lookup. Fallibility of map lookups is a separate problem that may require an independent solution, tracked in #27521. This issue is only about cost. ### Describe the solution you'd like Make `get_field` report a placement that reflects the cost of a map lookup, which is not `MoveTowardsLeafNodes`. The difficulty is that the decision needs types, and `placement()` has none. Some options: 1. **Pass argument types to `placement`**, for example by adding the argument fields or data types to the call, or by adding a new method such as `placement_with_fields(&self, args: &[ExpressionPlacement], arg_fields: &[FieldRef])`. `ScalarFunctionExpr` and `Expr::ScalarFunction` would then supply the types. The logical `Expr::placement` has no schema in scope, so it would need a schema parameter or a different entry point. This is the most general option, but it changes a public trait method (see the API health policy). 2. **Decide in the callers.** The optimizer rules and physical expressions that look at placement already work on plans that have schemas. They could ask for the placement with a schema in hand, and `get_field` would implement the type-aware method. 3. **Resolve at planning time.** Where the types are known (for example in `return_field_from_args` or a simplification), `get_field` could be rewritten to a map-specific function or marked in a way that its placement can read. This avoids changing the placement signature, but adds a second representation of map lookups. I think option 1 or 2 is the most robust. Whichever is chosen, `get_field` should return `KeepInPlace` when any step of its access path reaches into a `Map`, and keep `MoveTowardsLeafNodes` for struct-only paths. ### Describe alternatives you've considered - **Leave `get_field` as it is** and have each consumer special-case maps. #26589 does this for the physical `CASE` evaluation (`crosses_map` in `datafusion/physical-expr/src/expressions/case.rs`), but the logical optimizer rules do not, and every new consumer of placement would need to repeat it. - **Document that `MoveTowardsLeafNodes` may be assigned to expressions with data-dependent cost.** This makes the hint less useful for the decisions it exists to support. ### Additional context - Introduced in the discussion of #26589 (https://github.com/apache/datafusion/pull/26589#discussion_r4581490634). - Related: #27487, which asks for the speculative-evaluation requirement of `MoveTowardsLeafNodes` to be documented. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
