jaredmdobson opened a new issue, #70447:
URL: https://github.com/apache/doris/issues/70447
### Search before asking
- [x] I searched the issues and found no similar issues.
### Version
4.1.5 (apache/doris:fe-4.1.5 / be-4.1.5 images). The same code is still on
`branch-4.1`.
### What's wrong?
Planning a query fails with `errCode = 2, detailMessage =
java.lang.NullPointerException` when a projection that holds a lambda
(`array_filter(x -> ...)`, `array_map(x -> ...)`) sits above a chain of outer
joins and column statistics exist. `SELECT`, `EXPLAIN` and `CREATE VIEW` all
fail.
```
Caused by: java.lang.NullPointerException
at
com.google.common.base.Preconditions.checkNotNull(Preconditions.java:906)
at
com.google.common.collect.ImmutableList$Builder.add(ImmutableList.java:841)
at org.apache.doris.nereids.trees.TreeNode.rewriteUp(TreeNode.java:194)
...
at
org.apache.doris.nereids.rules.exploration.join.PushDownProjectThroughInnerOuterJoin.lambda$pushdownProject$13(PushDownProjectThroughInnerOuterJoin.java:123)
at
org.apache.doris.nereids.rules.exploration.join.PushDownProjectThroughInnerOuterJoin.pushdownProject(PushDownProjectThroughInnerOuterJoin.java:125)
at
org.apache.doris.nereids.rules.exploration.join.PushDownProjectThroughInnerOuterJoin.lambda$buildRules$3(PushDownProjectThroughInnerOuterJoin.java:69)
at
org.apache.doris.nereids.jobs.cascades.ApplyRuleJob.execute(ApplyRuleJob.java:82)
...
at
org.apache.doris.nereids.trees.plans.commands.info.BaseViewInfo.validate(BaseViewInfo.java:196)
at
org.apache.doris.nereids.trees.plans.commands.CreateViewCommand.run(CreateViewCommand.java:41)
```
Cause: for an outer join, `pushdownProject` rewrites every `Slot` in the
projections through `childrenSlots` (the join children's output):
```java
projects = project.getProjects().stream().map(expr -> expr.rewriteUp(e ->
e instanceof Slot ? childrenSlots.get((Slot) e) : e
)).map(e -> (NamedExpression) e).collect(Collectors.toList());
```
The lambda argument is an `ArrayItemReference.ArrayItemSlot`
(`SlotNotFromChildren`). It is not a child output, so `get` returns null and
`rewriteUp` NPEs. The hyper-edge check above it passes, because
`getInputSlotExprIds()` leaves `SlotNotFromChildren` slots out. Suggested fix:
keep slots that are not in the map, e.g. `childrenSlots.getOrDefault((Slot) e,
(Slot) e)`, or skip `SlotNotFromChildren`.
### What you expected to happen
The query plans and returns `[1, ["p", "z"]]`, `[2, NULL]`.
### How to reproduce
```sql
DROP DATABASE IF EXISTS lambda_npe FORCE;
CREATE DATABASE lambda_npe;
USE lambda_npe;
CREATE TABLE a (id INT) DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1
PROPERTIES ("replication_num" = "1");
CREATE TABLE b (id INT, arr ARRAY<STRING>) DUPLICATE KEY(id) DISTRIBUTED BY
HASH(id) BUCKETS 1 PROPERTIES ("replication_num" = "1");
CREATE TABLE c (id INT, w STRING) DUPLICATE KEY(id) DISTRIBUTED BY HASH(id)
BUCKETS 1 PROPERTIES ("replication_num" = "1");
CREATE TABLE d (id INT) DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1
PROPERTIES ("replication_num" = "1");
CREATE TABLE e (id INT) DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1
PROPERTIES ("replication_num" = "1");
INSERT INTO a VALUES (1), (2);
INSERT INTO b VALUES (1, ['p', '']);
INSERT INTO c VALUES (1, 'z');
INSERT INTO d VALUES (1);
INSERT INTO e VALUES (1);
ANALYZE TABLE a WITH SYNC;
ANALYZE TABLE b WITH SYNC;
ANALYZE TABLE c WITH SYNC;
ANALYZE TABLE d WITH SYNC;
ANALYZE TABLE e WITH SYNC;
SELECT a.id, array_filter(x -> x != '', array_concat(b.arr, array(c.w))) AS f
FROM a
LEFT JOIN b ON b.id = a.id
LEFT JOIN d ON d.id = a.id
LEFT JOIN c ON c.id = a.id
LEFT JOIN e ON e.id = a.id;
```
Without the `ANALYZE` statements the query plans fine (left-zig-zag rules
apply instead). `SET
disable_nereids_rules='PUSH_DOWN_PROJECT_THROUGH_INNER_OUTER_JOIN_LEFT,PUSH_DOWN_PROJECT_THROUGH_INNER_OUTER_JOIN_RIGHT'`
or `SET disable_join_reorder=true` avoids it.
### Anything Else?
#67541 removed this rule from master; `branch-4.1` still has it.
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's Code of Conduct
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
