andygrove commented on PR #6907: URL: https://github.com/apache/datafusion-comet/pull/6907#issuecomment-6151609695
The benchmark shows that when rows match many rows, the native sort-merge join is slower than what Comet does today with the join left in Spark: twice the time at 100 matches per row, and 2.7x at 1000. That isn't specific to `Cross`. An `Inner` join of the same shape gets the same native plan and costs the same. Profiling put most of the native time in DataFusion's sort-merge join recording its matches one buffered row at a time, so I opened apache/datafusion#26277 to append them in bulk. With a prototype of that change on DataFusion 55.2.0, measured in the same runs: | Matches per row | Native SMJ vs. join in Spark, now | With the fix | | --- | --- | --- | | 10 | 1.07x | 0.90x | | 100 | 2.0x | 1.0x | | 1000 | 2.7x | 1.4x | That cuts the native join's CPU time by 39% at 100 matches per row and 46% at 1000. What's left at very high fan-out is mostly building the output batches. The join gathers every column of both sides because it has no built-in projection, and its batch coalescer copies them again (apache/datafusion#25515). I'll keep this PR in draft until the fix reaches Comet, then rerun `CometCrossJoinBenchmark` and update the numbers here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
