Doris-Breakwater commented on issue #70448: URL: https://github.com/apache/doris/issues/70448#issuecomment-6773078466
Breakwater-GitHub-Analysis-Slot: slot_a747bd49d3c7 This report warrants FE/Nereids query-correctness triage for the reported 4.1.5 build. The expected unmatched-row values are consistent with evaluating the SELECT expressions after LEFT JOIN null extension. The issue currently has no labels. Source inspection identifies a concrete unsafe pushdown path; attribution to the running containers still needs the targeted check below. I have not independently executed the reproducer. The following is verified against the public `4.1.5-release` source, commit `917b20be83878e98f94e3ebf7efccc80346e3e50`: - [`ArrayMap`](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayMap.java#L40-L41) implements `PreferPushDownProject`. [`ArrayFilter`'s lambda constructor](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayFilter.java#L43-L48) rewrites `array_filter(lambda, A)` into `array_filter(A, array_map(lambda, A))`. Thus pushing only the predicate's internal `array_map` is enough to explain the filtered result; the entire `array_filter` need not move below the join. - [`Expression.getInputSlots()` excludes `ArrayItemSlot`](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/Expression.java#L73-L74), while traversal still sees `b.arr` inside the lambda's array argument. Ignoring the lambda-local `x` correctly identifies the external dependency as `b.arr`; that alone does not establish that pushing the expression across an outer join is safe. - [`PushDownProject.PushdownProjectHelper.pushDownExpression`](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownProject.java#L383-L405) selects a child using `child.getOutputSet().containsAll(e.getInputSlots())`, creates an alias below that child, and replaces the original expression with its slot. This path has no join-type or null-input-result guard. The SELECT-project entry is registered as `PUSH_DOWN_PROJECT_THROUGH_JOIN` and is gated by `enablePruneNestedColumns`. This mechanism explains both symptoms: for `a.id = 2`, no right-side row exists on which to evaluate the pushed map. Its alias becomes SQL NULL through join null extension, rather than evaluating the original expression on `b.arr = NULL` to produce `["N"]`. For filtering, the pushed predicate-mask alias can likewise become NULL even while the array argument above the join becomes `["n"]`; the BE array filter's [default NULL propagation](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/be/src/exprs/function/function.h#L136-L143) then yields NULL. The relationship to [#70447](https://github.com/apache/doris/issues/70447) should remain separate. That report concerns an unchecked lambda-slot lookup in `PushDownProjectThroughInnerOuterJoin`; the wrong-result candidate here is `PushDownProject`. Fixing the former's NPE or disabling only its two exploration rules does not establish a fix for this result. There is directly relevant public work: [PR #67889](https://github.com/apache/doris/pull/67889) adds null-input safety checks to this same helper. It is **open and unmerged** at the time of this review. Its project-pushdown test additions cover outer-join null semantics using other expressions, but do not explicitly cover `array_map`/`array_filter`. Treat it as a candidate fix to validate with this reproducer, not an available released fix. For confirmation, please attach: 1. Full `EXPLAIN VERBOSE` for the supplied SELECT, including projection expressions and slot IDs. The shape plan alone cannot identify which expression or internal mask moved. 2. Results and the verbose plan from the same query with the following rule disabled in a fresh session, or appended to the existing disabled-rule list: ```sql SET disable_nereids_rules = 'PUSH_DOWN_PROJECT_THROUGH_JOIN'; ``` If this is the responsible path, the maps should remain above the join and row 2 should become `plain = ["n"]`, `filtered = ["n"]`, `mapped = ["N"]`. This is a source-grounded diagnostic and possible session workaround, not a runtime-verified workaround. Close the diagnostic session afterward. 3. Exact FE/BE build versions/commit identifiers or image digests, plus the original `disable_nereids_rules` and `enable_prune_nested_column` values, to match the deployed images to the inspected source. For maintainers, retain post-join expressions on the null-supplying side unless the complete expression is proven to evaluate to SQL NULL on null-extended inputs; incomplete evaluation must not authorize pushdown. Preserve lambda-local slot handling. Add result and expression-placement coverage for this exact reproducer, matched rows with NULL/empty arrays and NULL elements, and RIGHT/FULL OUTER JOIN variants, including the internal filter mask. Confirm both original lambda forms against #67889 before deciding whether to extend that PR and backport to 4.1. No repository code was changed during this investigation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
