comphead commented on issue #4361: URL: https://github.com/apache/datafusion-comet/issues/4361#issuecomment-5039103308
according to AWS report https://awslabs.github.io/data-on-eks/docs/benchmarks/spark-datafusion-comet-benchmark#comet-underperforms-on-one-query ``` Comet still won on the systemic metrics — GC 38,079 ms → 72 ms and its scan stages were faster (36.6 s → 23.6 s) — but the loss is isolated to a single stage: the large store_sales ⋈ store_returns sort-merge-join + sort reducer. That stage ran ~2× slower under Comet (CometSortMergeJoin + CometSort, 90.0 s) than Spark's whole-stage-codegen SMJ + Sort (45.9 s). The +44 s from the slower native join/sort outweighs the −13 s saved on scans, netting +31 s. Root cause: Comet's native sort-merge-join + sort path is currently ~2× slower than Spark's JIT-compiled codegen for this large fact-to-fact join, and that single stage dominates q50 — so its IO/GC savings cannot compensate. ``` In 0.17 there were SMJ improvements possibly leading Q50 to perform better. Need to wait next AWS report based on 0.17 or 1.0.0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
