gene-bordegaray opened a new issue, #23376:
URL: https://github.com/apache/datafusion/issues/23376
# Support dynamic filters for range co-partitioned joins
Labels: `enhancement`
Related:
- #20195
- #21207
- #21992
- #22395
- #23184
## Is your feature request related to a problem or challenge?
Partitioned hash join dynamic filter pushdown is currently conservative for
non-hash partitioning. That is correct for now: partition-routed dynamic
filters
depend on stable partition identity, and incompatible routing can incorrectly
filter rows from the probe side.
With range co-partitioned joins, the optimizer can prove a stronger property:
left partition `i` and right partition `i` cover compatible key domains. In
that case, the build-side dynamic filter for partition `i` can potentially be
routed only to probe partition `i`, just as hash/hash partitioned joins do
today.
## Describe the solution you'd like
Allow partitioned join dynamic filters when the join inputs are proven
co-partitioned by the join keys, including compatible range/range
partitioning.
The implementation should:
- Reuse the co-partitioning proof used by partitioned join distribution
requirements.
- Allow range/range inputs only when partition counts, split points, ordering
options, and join-key requirements are compatible.
- Keep the current safe fallback when co-partitioning cannot be proven.
- Ensure dynamic filters are routed to the matching probe partition only.
- Add tests for compatible range/range joins, incompatible range boundaries,
and
hash/hash behavior remaining unchanged.
## Describe alternatives you've considered
The current alternative is to reject partitioned dynamic filter pushdown for
range partitioning. This is safe but leaves performance on the table for
already co-partitioned range inputs.
Another option is a global dynamic filter that is applied to all probe
partitions. That is simpler, but less precise and potentially more expensive
than partition-local routing.
## Additional context
This should remain separate from #23184. That PR should establish correctness
for avoiding unnecessary hash repartitioning on compatible range inputs.
Dynamic
filter routing can build on that requirement model once the co-partitioning
contract has settled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
