Doris-Breakwater commented on issue #70002: URL: https://github.com/apache/doris/issues/70002#issuecomment-6582015888
Breakwater-GitHub-Analysis-Slot: slot_145bc3397a97 **Triage:** This belongs with FE/Nereids analysis and subquery rewriting. The supplied reproduction and stack trace support a planner failure with inadequate diagnostics. Source inspection identifies a concrete mechanism consistent with the NPE; I have not executed the SQL on the reporter's backported build, so this is a source-backed diagnosis, not a runtime-confirmed root cause. The issue currently has no labels; `bug` and the project's applicable planner/component label would be appropriate. **Verified source behavior** I inspected the public `4.1.4` tag (`4accb51bc1f27d84980668f3a2eaf0ac96fb3cba`) and current master at `4bc7f426828e4cedfd2b7ce9b76fe95b14b5b146`, without changing the repository. 1. In 4.1.4, `ExpressionAnalyzer.processHighOrderFunction` analyzes the array inputs separately, creates lambda argument slots, and analyzes the lambda body without a lambda-specific subquery rejection. `SubqueryToApply` recursively collects subqueries from projection expressions and builds relational `LogicalApply` nodes below the projection. Its collector does not stop at a lambda boundary. See [lambda binding](https://github.com/apache/doris/blob/4accb51bc1f27d84980668f3a2eaf0ac96fb3cba/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/ExpressionAnalyzer.java#L400-L432) and [subquery collection](https://github.com/apache/doris/blob/4accb51bc1f27d84980668f3a2eaf0ac96fb3cba/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/SubqueryToApply.java#L734-L745). 2. `InApplyToJoin` constructs `EqualTo(compareExpr, subqueryOutput)`. Here the comparison input can be the lambda's `ArrayItemSlot`. Crucially, this slot is translated through **`findColumnRef`**, not the ordinary `findSlotRef` path. The column-reference mapping is installed during `visitArrayMap`, before translating its lambda body. A comparison moved into a child join can be translated before that mapping exists: `visitPhysicalProject` translates its child first. A missing mapping returns null, which reaches the non-null operand checks in `BinaryPredicate`. This is consistent with the reported stack and explains why the problem can affect constant arrays as well as column arrays. See [IN-to-join conversion](https://github.com/apache/doris/blob/4accb51bc1f27d84980668f3a2eaf0ac96fb3cba/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/InApplyToJoin.java#L94-L117), [lambda slot lookup](https://github.com/apache/doris/blob/4accb51bc1f27d84980668f3a2eaf0ac96fb3cba/fe/fe -core/src/main/java/org/apache/doris/nereids/glue/translator/ExpressionTranslator.java#L335-L342), and [mapping initialization](https://github.com/apache/doris/blob/4accb51bc1f27d84980668f3a2eaf0ac96fb3cba/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/ExpressionTranslator.java#L565-L606). 3. For `EXISTS (... WHERE v = x)`, the subquery analyzer can bind `x` from its immediate outer lambda scope. Correlated EXISTS rewriting uses the correlation filter as a join condition, providing another route for a lambda-local reference to escape into a relational predicate. This is consistent with the same failure family, but the complete stack and rewritten plan are needed to verify the exact path for that statement. See [correlated EXISTS conversion](https://github.com/apache/doris/blob/4accb51bc1f27d84980668f3a2eaf0ac96fb3cba/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/ExistsApplyToJoin.java#L94-L110). 4. The `t.s` case fails earlier, during name binding. In 4.1.4 the subquery's immediate outer scope contains the lambda arguments; the table scope is another level out. Ordinary slot binding searches the local scope and only one outer scope. Thus the error does not establish that `t.s` is absent from the table. Treat this as a separate scope/diagnostic case, not proof that it reached the NPE path. See [one-level outer lookup](https://github.com/apache/doris/blob/4accb51bc1f27d84980668f3a2eaf0ac96fb3cba/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/ExpressionAnalyzer.java#L281-L301) and [subquery scope construction](https://github.com/apache/doris/blob/4accb51bc1f27d84980668f3a2eaf0ac96fb3cba/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/SubExprAnalyzer.java#L233-L246). 5. The blank detail is also consistent with the exception path: the relevant `BinaryPredicate` constructor uses message-free null checks, and the query-planning catch wraps `e.getMessage()` in `AnalysisException`. A null exception message therefore loses useful context at this boundary. See [operand checks](https://github.com/apache/doris/blob/4accb51bc1f27d84980668f3a2eaf0ac96fb3cba/fe/fe-core/src/main/java/org/apache/doris/analysis/BinaryPredicate.java#L143-L153) and [planning exception wrapping](https://github.com/apache/doris/blob/4accb51bc1f27d84980668f3a2eaf0ac96fb3cba/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java#L825-L833). Current master still has the relevant unrestricted subquery collector and lambda-column lookup mechanism. Its direct lambda name binding has evolved, so the complete reproduction matrix should be tested there independently; source similarity alone does not prove identical runtime behavior or establish a fixed release. See [current lambda binding](https://github.com/apache/doris/blob/4bc7f426828e4cedfd2b7ce9b76fe95b14b5b146/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/ExpressionAnalyzer.java#L451-L503) and [current subquery collector](https://github.com/apache/doris/blob/4bc7f426828e4cedfd2b7ce9b76fe95b14b5b146/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/SubqueryToApply.java#L1058-L1067). **Information needed to confirm** - Exact FE build commit and the branch-4.1 backport list, plus `SELECT VERSION()` output. The release number alone cannot identify this build's behavior. - The complete FE exception chain for each failing statement, with query ID/timestamp and relevant non-default planner/session settings. - If obtainable, an analyzed/rewritten plan or a sanitized Nereids minidump for one IN and one correlated EXISTS example. The key check is whether an `ArrayItemSlot` appears in a relational join predicate with no corresponding column-reference mapping. If EXPLAIN also fails, its full exception is useful; a BE execution profile is not necessary to establish this planning failure. **Recommended next steps** - Reproduce the supplied matrix on the exact build and clean upstream revisions. Add regression coverage for IN/NOT IN, EXISTS/NOT EXISTS, scalar correlation to the lambda argument and to an outer table column, constant/column arrays, and nested lambdas. Include empty inputs, NULL elements/RHS values, and duplicate array elements so any eventual support preserves per-element and SQL null semantics. - For an initial diagnostic fix, reject unsupported subquery shapes in the **lambda body** before relational extraction. Preserve the reported working array-argument subqueries and uncorrelated scalar-body subqueries. A blanket `SubqueryExpr` rejection under the entire lambda tree would also catch array inputs stored in lambda argument references and would be too broad. Correlation to `t.s` needs an intentional scope-aware diagnostic before ordinary binding throws; checking correlation metadata only after successful analysis would miss this case. - Do not address this solely by fabricating a missing translation mapping: a relational join predicate cannot acquire per-array-element semantics that way. Full support needs a deliberate element-level rewrite. Separately, preserve a non-empty diagnostic when a planning exception has no message. - The two SQL alternatives in the report are useful starting points for the supplied non-null membership example, but are not interchangeable general replacements. The EXPLODE query returns matching IDs only and `DISTINCT` collapses duplicates; it does not return one Boolean result for every input row or preserve filtered-array order/multiplicity. Validate NULL and empty-input behavior before replacing other shapes with ARRAYS_OVERLAP. No code changes were made and no runtime reproduction is claimed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
