haohuaijin opened a new issue, #25264:
URL: https://github.com/apache/datafusion/issues/25264
### Describe the bug
Enabling join dynamic filter pushdown can silently discard a matching row
when the filter passes through an `AggregateExec` that groups on two columns
with the same name (here `a.id` and `b.id` from a nested join).
With the same data and SQL, disabling
`datafusion.optimizer.enable_join_dynamic_filter_pushdown` returns one row;
enabling it returns zero rows, without an error.
### To Reproduce
Run the following SQL in a fresh `datafusion-cli` session. The two `/tmp/`
Parquet paths must not already exist.
```sql
SET datafusion.optimizer.join_reordering = false;
COPY (SELECT 'a1' AS id, 'x1' AS ty)
TO '/tmp/df_dynamic_filter_aggregate_a.parquet'
STORED AS PARQUET;
COPY (SELECT 'x1' AS id)
TO '/tmp/df_dynamic_filter_aggregate_b.parquet'
STORED AS PARQUET;
CREATE EXTERNAL TABLE ta
STORED AS PARQUET
LOCATION '/tmp/df_dynamic_filter_aggregate_a.parquet';
CREATE EXTERNAL TABLE tb
STORED AS PARQUET
LOCATION '/tmp/df_dynamic_filter_aggregate_b.parquet';
-- Returns one row: (x1, a1, x1).
SET datafusion.optimizer.enable_join_dynamic_filter_pushdown = false;
SELECT s.id, a.id, b.id
FROM tb s
JOIN (
SELECT a.id, b.id
FROM ta a LEFT JOIN tb b ON a.ty = b.id
GROUP BY a.id, b.id
)
ON s.id = b.id;
-- Incorrectly returns zero rows.
SET datafusion.optimizer.enable_join_dynamic_filter_pushdown = true;
SELECT s.id, a.id, b.id
FROM tb s
JOIN (
SELECT a.id, b.id
FROM ta a LEFT JOIN tb b ON a.ty = b.id
GROUP BY a.id, b.id
)
ON s.id = b.id;
```
### Expected behavior
_No response_
### Additional context
`AggregateExec` resolves pushed-down filter columns by name instead of by
grouping position, so the filter on the grouping column `b.id` is applied to
`a.id`. Found while working on #25244; the fix is included in #25259.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
