jayzhan211 opened a new issue, #25708:
URL: https://github.com/apache/datafusion/issues/25708
### Is your feature request related to a problem or challenge?
Follow-up to #25529 (fix for #25519).
To stop wrong results, #25529 keeps a correlated filter below an aggregate
with a grouping set unless every set already groups by each column the pull up
would add. That guard is conservative: it also rejects non-empty sets that
leave out the correlated column. On `main` those queries were answered
correctly, and with #25529 they fail to plan:
```sql
CREATE TABLE o(k INT) AS VALUES (1), (2), (NULL), (4), (5);
CREATE TABLE i(k INT, j INT) AS VALUES (1, 10), (NULL, 20), (5, 30), (2, 40);
SELECT o.k FROM o
WHERE EXISTS (
SELECT 1 FROM i WHERE i.k = o.k
GROUP BY GROUPING SETS ((i.k), (i.j))
)
ORDER BY o.k;
-- with #25529: This feature is not implemented: Physical plan does not
support logical expression Exists(...)
```
Because the filter `i.k = o.k` fixes `i.k` to one value per outer row,
adding `i.k` to a non-empty set that lacks it leaves the rows of each outer row
unchanged. Only two things change: the value of `i.k` in those rows (NULL
becomes `o.k`) and `__grouping_id`. So the old rewrite was wrong only when
something above the aggregate reads the NULL-filled column or `GROUPING()`. One
such case is `HAVING i.k IS NULL`, which is covered in `subquery.slt`.
### Describe the solution you'd like
Decorrelate these queries again without bringing back the wrong results. Two
options:
1. Add the correlated column to each set that lacks it under an alias, so
the column the query reads keeps its NULL fill and the join key is a separate
column.
2. Reject only when a set is empty (`ROLLUP`/`CUBE` always contain one) or
when a node above the aggregate reads the NULL-filled column or `GROUPING()`.
Empty sets must stay rejected: they yield a row for outer rows that match
nothing, and a join cannot produce that row.
### Describe alternatives you've considered
Keep the current guard. The queries fail to plan instead of returning wrong
results.
### Additional context
The guard is in the `Aggregate` arm of `PullUpCorrelatedExpr::f_up`,
`datafusion/optimizer/src/decorrelate.rs`. The known limitation is documented
in `datafusion/sqllogictest/test_files/subquery.slt`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
