nuno-faria opened a new issue, #20214:
URL: https://github.com/apache/datafusion/issues/20214
### Is your feature request related to a problem or challenge?
I would like to enable dynamic filter pushdown for inner joins which use
other joins as subqueries. For example:
```sql
select *
from (
select *
from t1
left anti join t2 on t1.k = t2.k
) a
join t1 b on a.k = b.k
where b.v = 1;
```
Even though a dynamic filter is created (`DynamicFilter [ k@0 >= 1 AND k@0
<= 1 AND k@0 IN (SET) ([1]) ]`), it is not pushed to subquery `a`.
### Describe the solution you'd like
I think the issue is that `gather_filters_for_pushdown` is used to both
build dynamic filters and determine if they can be applied. Currently, the
dynamic filters are dropped for all join types apart from `inner` ones, which
explains why it is not pushed to the subquery above which uses a `left anti`
join. I tried locally and this can be solved by making
`gather_filters_for_pushdown` not disallowing filters, but still preventing
them from being generated from non-inner joins which is known to cause issues
(see https://github.com/apache/datafusion/issues/16973).
I think the only potential issue might be when we use `left`/`right` joins
with duplicate column names, which could cause the filter to be pushed to the
wrong table. But I found out that this also happens right now with `inner`
joins, so I opened #20213 to track it. Anti joins are, however, completely safe
in any case.
@adriangb what do you think? Does this make sense, or am I missing some edge
case?
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
