matanper opened a new issue, #70002: URL: https://github.com/apache/doris/issues/70002
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version 4.1.4 (built with some branch-4.1 backports), Nereids planner. I found no check for this case on master either. ### What's Wrong? An `IN (SELECT …)` or `EXISTS (SELECT …)` predicate inside a lambda body fails in the FE with a `NullPointerException`. The client gets `ERROR 1105` with an **empty** `detailMessage`, so the user can't tell what's wrong. FE log: ``` java.lang.NullPointerException at org.apache.doris.analysis.BinaryPredicate.<init>(BinaryPredicate.java:147) at org.apache.doris.nereids.glue.translator.ExpressionTranslator.visitEqualTo(ExpressionTranslator.java:163) at org.apache.doris.nereids.glue.translator.ExpressionTranslator.visitEqualTo(ExpressionTranslator.java:135) ... ``` Other subquery shapes around lambdas do work, so the failure is specific to a predicate subquery in the lambda body: | Shape | Result | |---|---| | Subquery as the lambda's array argument: `ARRAY_MAP(x -> UPPER(x), (SELECT ARRAY_AGG(v) FROM ioc))` | works | | Uncorrelated scalar subquery in the body: `ARRAY_MAP(x -> CONCAT(x, (SELECT MAX(v) FROM ioc)), ['a','b'])` | works | | `IN` subquery in the body, constant array: `ARRAY_FILTER(x -> x IN (SELECT v FROM ioc), ['a','b'])` | NPE, empty error | | `IN` subquery in the body, column array: `ARRAY_EXISTS(x -> x IN (SELECT v FROM ioc), a)` / `ARRAY_COUNT(...)` | NPE, empty error | | `EXISTS` correlated to the lambda variable: `ARRAY_EXISTS(x -> EXISTS (SELECT 1 FROM ioc WHERE v = x), a)` | NPE, empty error | | Scalar subquery in the body correlated to an outer column: `ARRAY_MAP(x -> CONCAT(x, (SELECT MAX(v) FROM ioc WHERE v > t.s)), a)` | `Unknown column 's' in 't' in FILTER clause` | No BE was affected; the failure is entirely in the FE. ### What You Expected? Either support these shapes, or reject them at analysis time with a clear message, e.g. `subquery predicate inside a lambda is not supported`, instead of an NPE in the expression translator. The last row's error is also misleading, because column `s` does exist on `t`: a correlated reference from a subquery inside a lambda is what's unsupported. ### How to Reproduce? ```sql CREATE TABLE t (id INT, s STRING, a ARRAY<STRING>) DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ('replication_num' = '1'); CREATE TABLE ioc (v VARCHAR(255)) DUPLICATE KEY(v) DISTRIBUTED BY HASH(v) BUCKETS 1 PROPERTIES ('replication_num' = '1'); INSERT INTO t VALUES (1, 'a', ['evil.com', 'ok.com']), (2, 'b', ['ok.com']), (3, 'c', []); INSERT INTO ioc VALUES ('evil.com'), ('bad.net'); -- ERROR 1105 (HY000): errCode = 2, detailMessage = (NPE in the FE log) SELECT id, ARRAY_EXISTS(x -> x IN (SELECT v FROM ioc), a) FROM t ORDER BY id; SELECT id, ARRAY_EXISTS(x -> EXISTS (SELECT 1 FROM ioc WHERE v = x), a) FROM t ORDER BY id; SELECT ARRAY_FILTER(x -> x IN (SELECT v FROM ioc), ['evil.com', 'ok.com']); ``` Equivalent queries that work today: ```sql SELECT id, ARRAYS_OVERLAP(a, (SELECT ARRAY_AGG(v) FROM ioc)) FROM t ORDER BY id; SELECT DISTINCT t.id FROM t LATERAL VIEW EXPLODE(t.a) e AS x WHERE x IN (SELECT v FROM ioc) ORDER BY t.id; ``` ### Anything Else? `ExpressionAnalyzer` binds the lambda body without checking for `SubqueryExpr`. I haven't traced the rest, but the stack suggests the predicate subquery gets rewritten outside the lambda, leaving an `EqualTo` whose lambda-argument side the translator can't resolve. A check that rejects `InSubquery` / `Exists` (and correlated subqueries) under a `Lambda` during analysis would turn this into a clear error. ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
