Doris-Breakwater commented on issue #69930: URL: https://github.com/apache/doris/issues/69930#issuecomment-6563965715
This is actionable as an **FE/Nereids planner correctness bug**. Source inspection supports the reported lambda-scope violation; the NPE is a downstream symptom. The issue currently has no labels. Suggested routing: Nereids optimizer / expression pushdown. I checked `branch-4.1` at `b4afc8c693ef6fbb2a657f0a5fcbcd14e52a30d2` and `master` at `8e10b235cf030d486b6840cfd64b6218ab515e6d` (both matched the public branch heads when checked). The relevant unsafe eligibility checks exist in both. This was a read-only source investigation: I did not execute the SQL, build Doris, or modify product code, so the runtime outcomes below remain reporter-provided evidence. The code confirms the following chain: 1. `STRUCT_ELEMENT` is registered as an alias of `ElementAt`, which implements `PreferPushDownProject`. `ArrayFirst` also constructs an `ArrayMap` for its predicate. Thus the field access inside the lambda is eligible for this traversal. See [function registration](https://github.com/apache/doris/blob/b4afc8c693ef6fbb2a657f0a5fcbcd14e52a30d2/fe/fe-core/src/main/java/org/apache/doris/catalog/BuiltinScalarFunctions.java#L774) and [ArrayFirst](https://github.com/apache/doris/blob/b4afc8c693ef6fbb2a657f0a5fcbcd14e52a30d2/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayFirst.java#L50). 2. [`Expression.getInputSlots()` excludes `ArrayItemSlot`](https://github.com/apache/doris/blob/b4afc8c693ef6fbb2a657f0a5fcbcd14e52a30d2/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/Expression.java#L73). Consequently, the isolated `element_at(p, 'col1')` has no relational input slots, although it still requires a lambda binding for `p`. 3. [`PushdownProjectHelper.pushDownExpression`](https://github.com/apache/doris/blob/b4afc8c693ef6fbb2a657f0a5fcbcd14e52a30d2/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownProject.java#L383) traverses downward when a larger expression cannot move. With `k` and the array supplied by opposite join children, the whole `ArrayMap` cannot move to either child. Traversal then reaches the field read; `containsAll(emptySet)` succeeds for the first child, extracting the field read outside its lambda. Moving it to the array-producing child would also be invalid: that child supplies the array, not an individual lambda argument binding. 4. [`visitArrayItemSlot`](https://github.com/apache/doris/blob/b4afc8c693ef6fbb2a657f0a5fcbcd14e52a30d2/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/ExpressionTranslator.java#L341) looks up a `ColumnRefExpr`; `visitArrayMap` registers those bindings before translating its lambda body. An extracted field read lacks that binding. This explains a null child reaching [`Expr.extractSlots`](https://github.com/apache/doris/blob/b4afc8c693ef6fbb2a657f0a5fcbcd14e52a30d2/fe/fe-core/src/main/java/org/apache/doris/analysis/Expr.java#L447), consistent with the reported grouped-query exception. The exact throw site of the ungrouped variant still needs its stack trace. The proposed fix direction is sound: require each movable expression to have **no free lambda argument references**, in addition to satisfying relational input dependencies. Use binding identity (`ExprId`) and lexical scope, including nested lambdas and captures. Preserve movement of a complete, self-contained lambda expression when its relational inputs are available. Globally adding lambda arguments to `getInputSlots()`, or rejecting every expression containing a lambda, would change broader semantics or unnecessarily disable valid pushdown. There are **two implementations of the eligibility check** in `PushDownProject`: the helper above and [`pushDownProjectInExpressions`](https://github.com/apache/doris/blob/b4afc8c693ef6fbb2a657f0a5fcbcd14e52a30d2/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownProject.java#L162), used for join conjuncts. Review both, preferably with a shared scope check. A translator null guard alone would leave the invalid rewrite intact. For the proposed PR, add regression coverage for the supplied query with/without grouping and with both array constructors; assert results and that the field read remains inside its lambda. Also cover a complete movable lambda, scalar-array controls, nested lambda captures, and the join-conjunct path. Exercise the shared helper's filter, union, window, and partition-top-N callers where applicable; these are coverage recommendations, not independently reproduced failures. The reported session workaround, `SET enable_prune_nested_column = false`, is consistent with the gate on this project-pushdown path. Keep it scoped to affected sessions and restore the prior value afterward. It disables nested-column pruning more broadly and is not a demonstrated workaround for every caller of this rule. To close the remaining validation gaps, please attach the exact 4.1 build/version and commit, full FE exception stacks for both variants, and the available rewritten-plan/EXPLAIN output with pruning enabled and disabled. The existing table-free reproduction is sufficient to start a regression test; no table dump or runtime profile is needed for this planning failure. An introducing commit, tested fix, and release/backport coverage are not yet established. Breakwater-GitHub-Analysis-Slot: slot_3ac1a30d5059 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
