andygrove opened a new issue, #4849: URL: https://github.com/apache/datafusion-comet/issues/4849
### What is the problem the feature request solves? According to https://awslabs.github.io/data-on-eks/docs/benchmarks/spark-datafusion-comet-benchmark, q50 was the only query to run slower with Comet enabled. This page has the following notes: - Comet still won on the systemic metrics — GC 38,079 ms → 72 ms and its scan stages were faster (36.6 s → 23.6 s) — but the loss is isolated to a single stage: the large store_sales ⋈ store_returns sort-merge-join + sort reducer. - That stage ran ~2× slower under Comet (CometSortMergeJoin + CometSort, 90.0 s) than Spark's whole-stage-codegen SMJ + Sort (45.9 s). The +44 s from the slower native join/sort outweighs the −13 s saved on scans, netting +31 s. - Root cause: Comet's native sort-merge-join + sort path is currently ~2× slower than Spark's JIT-compiled codegen for this large fact-to-fact join, and that single stage dominates q50 — so its IO/GC savings cannot compensate. ### Describe the potential solution _No response_ ### Additional context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
