stuhood commented on issue #25301: URL: https://github.com/apache/datafusion/issues/25301#issuecomment-5708827844
> what would you think about running some / sharing some benches in DFD? https://github.com/datafusion-contrib/datafusion-distributed/pull/730 is almost ready for review to add these on the `df-d` side. In ParadeDB, I added an optimizer rule to convert `CollectLeft` back to `Partitioned` for co-partitioned joins: can see the results in https://github.com/paradedb/paradedb/pull/6342/changes#diff-89dfce63e3efead59030ea110c105379e915189aaa282cac639adb2981da39f8 > I see the point here, what I am thinking is that a partitioned hash join is going to have more build tables The co-partitioned join ends up with a smaller hash table per partition: it also frequently is able to apply the [perfect hash join](https://github.com/apache/datafusion/pull/19411) optimization dynamically at runtime, because the join keys have narrow `min`/`max` bounds. > ...would it ever be the case that we keep the collect left but do not broadcast thus we don't have network overhead and don't risk having things like build / read amp? Not sure what you mean here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
