jayzhan211 commented on code in PR #25339:
URL: https://github.com/apache/datafusion/pull/25339#discussion_r4056392706
##########
datafusion/optimizer/src/decorrelate_predicate_subquery.rs:
##########
@@ -310,6 +336,26 @@ fn has_subquery(expr: &Expr) -> bool {
.unwrap()
}
+/// True when every subquery in `expr` is a non-negated `IN`/`EXISTS` reached
only
+/// through `AND`/`OR`.
+///
+/// `AND` and `OR` give TRUE only when an operand is TRUE, so a `Filter` on
such an
+/// expression keeps the same rows whether a mark is NULL or FALSE. The mark
join
+/// then does not need to be null-aware. `NOT`, `IS NULL`, `CASE` and a mark
that
+/// goes into a projection can tell NULL from FALSE, so they give `false` here.
+fn subqueries_only_positive(expr: &Expr) -> bool {
+ match expr {
+ Expr::BinaryExpr(BinaryExpr {
+ left,
+ op: Operator::And | Operator::Or,
+ right,
+ }) => subqueries_only_positive(left) &&
subqueries_only_positive(right),
+ Expr::InSubquery(InSubquery { negated, .. }) => !negated,
Review Comment:
I found another issue
`InSubquery` arm returns `!negated` without checking `expr` → a subquery
inside the `IN` value gets a non-null-aware mark, but its NULL/FALSE is
observable there. Regression from commit 6: base `edc936f38` returns 0 rows,
this PR returns `2, 5, NULL, NULL`.
Repro:
```sql
CREATE TABLE t1(id INT, z INT, w INT) AS VALUES (1,10,1), (2,20,1),
(NULL,30,2), (4,40,2), (5,NULL,1), (NULL,NULL,2);
CREATE TABLE t2(id INT, z INT, w INT) AS VALUES (1,5,1), (NULL,50,1),
(4,NULL,2), (NULL,NULL,2), (2,20,3);
CREATE TABLE tb(b BOOLEAN) AS VALUES (false);
-- expected: no rows
SELECT id FROM t1
WHERE ((id IN (SELECT t2.id FROM t2 WHERE t2.w = t1.w)) IN (SELECT b FROM
tb)) OR id = -1
ORDER BY id;
```
Fix (verified locally, touched slt files still pass); please add the repro
to `null_aware_mark_join.slt`:
```diff
- Expr::InSubquery(InSubquery { negated, .. }) => !negated,
+ Expr::InSubquery(InSubquery { expr, negated, .. }) => {
+ !negated && !has_subquery(expr)
+ }
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
