haohuaijin opened a new issue, #25262:
URL: https://github.com/apache/datafusion/issues/25262
### Describe the bug
Enabling join dynamic filter pushdown can silently discard a matching row
when the filter passes through a `FilterExec` that carries an embedded
projection over an input with duplicate column names (here `a.id` and `b.id`
from a nested join).
With the same data and SQL, disabling
`datafusion.optimizer.enable_join_dynamic_filter_pushdown` returns one row;
enabling it returns zero rows, without an error.
### To Reproduce
Run the following SQL in a fresh `datafusion-cli` session. The two `/tmp/`
Parquet paths must not already exist.
```sql
SET datafusion.optimizer.join_reordering = false;
COPY (SELECT 'a1' AS id, 'x1' AS ty)
TO '/tmp/df_dynamic_filter_filterprojection_a.parquet'
STORED AS PARQUET;
COPY (SELECT 'x1' AS id)
TO '/tmp/df_dynamic_filter_filterprojection_b.parquet'
STORED AS PARQUET;
CREATE EXTERNAL TABLE ta
STORED AS PARQUET
LOCATION '/tmp/df_dynamic_filter_filterprojection_a.parquet';
CREATE EXTERNAL TABLE tb
STORED AS PARQUET
LOCATION '/tmp/df_dynamic_filter_filterprojection_b.parquet';
-- Returns one row: (x1, a1, x1).
SET datafusion.optimizer.enable_join_dynamic_filter_pushdown = false;
SELECT s.id AS sid, a.id AS aid, b.id AS bid
FROM tb s
JOIN (
SELECT a.id, b.id
FROM ta a LEFT JOIN tb b ON a.ty = b.id
WHERE (a.ty || '!') IS DISTINCT FROM b.id
)
ON s.id = b.id;
-- Incorrectly returns zero rows.
SET datafusion.optimizer.enable_join_dynamic_filter_pushdown = true;
SELECT s.id AS sid, a.id AS aid, b.id AS bid
FROM tb s
JOIN (
SELECT a.id, b.id
FROM ta a LEFT JOIN tb b ON a.ty = b.id
WHERE (a.ty || '!') IS DISTINCT FROM b.id
)
ON s.id = b.id;
```
### Expected behavior
_No response_
### Additional context
`FilterExec` resolves pushed-down filter columns by name instead of through
its embedded projection, so the filter on `b.id` is applied to `a.id`. Found
while working on #25244; the fix is included in #25259.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
