dwsmith1983 commented on issue #2545: URL: https://github.com/apache/datafusion-comet/issues/2545#issuecomment-5890829135
The first piece, the size guard in `RewriteJoin`, is up as #6384. With `spark.comet.exec.forceShuffledHashJoin` on, a sort-merge join is rewritten only when the build side's estimate is under a limit, by default Spark's own `canBuildLocalHashMapBySize` rule (`autoBroadcastJoinThreshold` times the shuffle partition count). Larger joins keep the sort-merge join, still native, with the reason in the plan info. `spark.comet.exec.forceShuffledHashJoin.maxBuildSize` sets a fixed limit or turns it off, and the two hash join benchmark configs set it to -1 so they keep measuring every join. The second piece waits for apache/datafusion#25217. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
