Doris-Breakwater commented on issue #70447: URL: https://github.com/apache/doris/issues/70447#issuecomment-6773052193
Breakwater-GitHub-Analysis-Slot: slot_ffeb5ebf9161 **Assessment:** The reported stack trace and explanation are consistent with a concrete FE/Nereids expression-rewrite defect in `4.1.5-release`. The same unchecked lookup is present at the current `branch-4.1` commit inspected (`6acf3dad10bd59d4f645ae998734a22199b90013`). This is sufficient to route to the Nereids join-reorder maintainers. The issue currently has no labels; suggested classification is bug / FE / Nereids optimizer. **Verified from source** - In [`PushDownProjectThroughInnerOuterJoin.pushdownProject`](https://github.com/apache/doris/blob/d1cbb760d2680b17f8971c3bf396e94f1adbf32b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/PushDownProjectThroughInnerOuterJoin.java#L94-L147), the outer-join path builds its replacement map exclusively from the two children's outputs, then visits every expression node and replaces every `Slot` with `childrenSlots.get(...)`. There is no fallback for a lambda-local slot. This traversal happens **before** the checks that reject complex projections on a nullable join side, so even a transformation that would eventually be rejected can throw first. - [`ArrayItemReference.toSlot()`](https://github.com/apache/doris/blob/d1cbb760d2680b17f8971c3bf396e94f1adbf32b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/ArrayItemReference.java#L100-L136) produces an `ArrayItemSlot`, which implements `SlotNotFromChildren`. The lambda body can therefore contain a legitimate `Slot` that is absent from the join-child map. One precision point: [`Expression`'s input-slot collector](https://github.com/apache/doris/blob/d1cbb760d2680b17f8971c3bf396e94f1adbf32b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/Expression.java#L70-L77) specifically excludes `ArrayItemSlot`, rather than testing the general `SlotNotFromChildren` interface. Thus the hyper-edge guard can pass when the real column inputs belong to one immediate join child, while the later tree traversal still reaches the lambda variable. - The missing lookup returns `null`; the parent invocation of [`TreeNode.rewriteUp`](https://github.com/apache/doris/blob/d1cbb760d2680b17f8971c3bf396e94f1adbf32b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/TreeNode.java#L188-L198) adds that result to an `ImmutableList.Builder`. This explains the reported `checkNotNull -> Builder.add -> rewriteUp` exception chain. `array_filter` also constructs an `ArrayMap` around its lambda, so the mechanism applies to both functions mentioned. **Scope and remaining uncertainty** This is source verification, not an independently executed SQL reproduction; no Doris cluster test or patch validation was performed. The supplied DDL, data, statistics collection and stack trace are already useful for a regression test. The statistics dependency should be treated as a reported trigger for this particular plan shape, not a necessary condition for the defect. In the release [`RuleSet`](https://github.com/apache/doris/blob/d1cbb760d2680b17f8971c3bf396e94f1adbf32b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleSet.java#L258-L285), left-zig-zag also includes `OTHER_REORDER_RULES`, which registers this rule. Statistics can change exploration and which matching expressions are reached; skipping `ANALYZE` does not guarantee safety. Also, [#67541](https://github.com/apache/doris/pull/67541) removes the standalone factories from **classic** registration, retains their after-DPHyp registration, and moves normalization into a shared helper. At upstream master commit `9fb3e3a2a13f972a62cb999d5599fc8e8f17f306`, [`ProjectJoinReorderHelper.adjustProjectsNullable`](https://github.com/apache/doris/blob/9fb3e3a2a13f972a62cb999d5599fc8e8f17f306/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/ProjectJoinReorderHelper.java#L162-L180) still contains the equivalent unchecked `childSlots.get(...)`. That is evidence of the same unsafe mechanism, not proof that this exact SQL reproduces on master. Do not regard that PR or an upgrade to master alone as a verified fix. **Recommended next steps** 1. Add a failing regression using the supplied query with statistics, then verify `SELECT`, `EXPLAIN` and `CREATE VIEW`. Cover both left/right rule variants, `array_filter` and `array_map`, and preserve the expected unmatched-row `NULL` result. Include a case that must reject pushdown on the nullable side without throwing. 2. Preserve lambda-local slots during the outer-join slot adjustment. Skipping `SlotNotFromChildren` expresses the intended distinction; the suggested `getOrDefault(slot, slot)` also avoids this null result but has broader fallback behavior. Keep normal child-slot/nullability replacement and the outer-join safety checks intact. Review the shared helper on master as well as the standalone implementation on 4.1 before deciding fix/backport scope. 3. For temporary mitigation, the reported session-level disabling of `PUSH_DOWN_PROJECT_THROUGH_INNER_OUTER_JOIN_LEFT` and `PUSH_DOWN_PROJECT_THROUGH_INNER_OUTER_JOIN_RIGHT` directly prevents these standalone rules from running on 4.1. Preserve any existing disabled-rule entries. `disable_join_reorder=true` is a broader fallback: the release optimizer returns no join-exploration rules when it is set, which can affect plan quality. Restore prior session settings after use. 4. If maintainers cannot reproduce on a clean 4.1.5 environment, request the exact FE build commit/version output, the full FE exception with query ID, and session settings for disabled rules, join reordering, DPHyp and left-zig-zag. A successful `EXPLAIN` under the workaround would help compare plan shape. An execution profile is not required to establish this planning-time null lookup. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
