kevinwilfong opened a new pull request, #13567: URL: https://github.com/apache/gluten/pull/13567
## What changes are proposed in this pull request? I recently added an optimization in Velox's SortMergeJoin operator to only buffer the right side of the join and stream the left side, when the join type allows. This change allows us to take advantage of the optimization in Gluten by applying the same optimization used in HashJoin to swap the inputs if necessary and put the larger input on the left side of the join. Preserves Spark children, keys, join types and output metadata. Preserves skew joins and joins without known input sizes, and retains forced hash join precedence. Other backends keep their existing conversion behavior. ## How was this patch tested? Changing the TPCH benchmark to use sort merge joins, Q3 and Q7 are relevant. Performance is on par, but memory usage is dramatically lower (this is expected): <html><head></head><body><p><strong>Memory:</strong> representative native join <code>peakMemoryBytes</code> snapshots:</p> Join | Optimization disabled | Optimization enabled | Reduction -- | -- | -- | -- Q3: customer/orders | 576 KiB | 240 KiB | 58.3% Q7: supplier/lineitem | 864 KiB | 288 KiB | 66.7% </body></html> Added tests to cover the translation and execution. ## Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex GPT-6 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
