Dandandan opened a new issue, #19789:
URL: https://github.com/apache/datafusion/issues/19789
What might be a good thing to try:
Currently, a high part of the cost of the `Partitioned` join is
repartitioning the entire build side of the join - which currently copies all
of the columns twice(!), this makes this hash join slower if this side is
large/wide. We could avoid one copy in `RepartitionExec` (using arrow-rs
coalesce API when fully implemented), but not two.
`CollectLeft` avoids this cost at the right side, but the building phase is
single threaded, which greatly limits the parallelism in the query.
We should be able to do the hash % mod of the build side **during** the
join, avoiding the need for a `RepartitionExec` passing the indices of the
matching partitions to the left sides - which should greatly reduce the
overhead of the repartitioning.
When this is implemented, we might want to look at
`hash_join_single_partition_threshold` and
`hash_join_single_partition_threshold_rows` again which could be reduced to
make most joins run fully in parallel./
_Originally posted by @Dandandan in
https://github.com/apache/datafusion/issues/19761#issuecomment-3743995482_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
