Dandandan commented on PR #19761:
URL: https://github.com/apache/datafusion/pull/19761#issuecomment-3743995482
What might be a good thing to try:
Currently, a high part of the cost of the `Partitioned` join is
repartitioning the entire build side of the join - which currently copies all
of the columns twice(!), this makes this hash join slower if this side is
large/wide. We could avoid one copy in `RepartitionExec` (using arrow-rs
coalesce API when fully implemented), but not two.
`CollectLeft` avoids this cost at the right side, but is single threaded,
which greatly limits the parallelism in the query.
We should be able to do the hash % mod of the build side **during** the
join, avoiding the need for a `RepartitionExec` passing the indices of the
matching partitions to the left sides - which should greatly reduce the
overhead of the repartitioning.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
