andygrove opened a new issue, #1671: URL: https://github.com/apache/datafusion-comet/issues/1671
### What is the problem the feature request solves? Spark's join strategy planner chooses between Broadcast Hash Join, Shuffled Hash Join, and Sort Merge Join based primarily on the estimated `sizeInBytes` of the build-side input. When using Comet, the input to a join may come from a Comet Exchange, which benefits from columnar compression. This often results in a `sizeInBytes` estimate that is 2x smaller (or more) than what Spark would compute for the same plan using row-based execution. As a result, Spark may incorrectly choose a Broadcast Hash Join when the actual uncompressed size of the data would have triggered a Shuffled Hash Join or Sort Merge Join. This can lead to performance regressions or even OOM issues on the driver or executor. Proposal: We should consider adjusting the reported `sizeInBytes` for Comet's columnar exchanges to more accurately reflect the logical uncompressed size of the data. This would align better with Spark’s assumptions and avoid unintended join strategy changes. Options might include: - Applying a fixed multiplier to sizeInBytes in columnar exchange operators (e.g., 2x) - Using schema-aware size estimates to approximate row-based sizes - Providing a config flag to tune this behavior for benchmarking and tuning Impact: - More consistent join strategy decisions with and without Comet - Reduced risk of performance regressions due to unexpectedly chosen broadcast joins - Better alignment with Spark’s cost model ### Describe the potential solution _No response_ ### Additional context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
