andygrove commented on PR #7460: URL: https://github.com/apache/datafusion-comet/pull/7460#issuecomment-6609595606
TPC-DS benchmark of this PR (head `718f2d6cc`), one run. Setup: Spark 4.2.0 (Scala 2.13), TPC-DS SF 1000 Parquet (zstd), 103 queries, 2 iterations (mean), 32 executors x 16 cores on EKS with data in S3. The Spark properties are the same as in the Comet 1.2.0 run, apart from the Comet version. | Total time | Speedup over Spark | |---|---| | Spark 4.2.0: 994 s | | | Comet 1.2.0: 488 s | 2.04x | | This PR: 541 s | 1.84x | | Comet 1.1.0: 601 s | 1.65x | - Compared with Comet 1.2.0, this PR is 0.90x in total (541 s vs 488 s) and 0.90x per query in geomean. 45 of 103 queries are more than 10% slower and 12 are more than 10% faster. - Largest absolute slowdowns: q23b 30 s to 39 s, q88 10 s to 14 s, q24a 17 s to 20 s, q28 10 s to 12 s, q72 10.3 s to 12.4 s. Largest relative slowdowns are short queries: q56 1.0 s to 2.7 s, q94 2.9 s to 6.2 s, q58 1.1 s to 2.0 s. - Largest gains: q54 6.3 s to 4.1 s, q47 5.8 s to 4.4 s, q01 6.4 s to 5.5 s, q68, q07, q08. - The S3-heavy queries did not improve: q30 unchanged (1.8 s), q09 9.6 s to 10.4 s, q88 10.2 s to 13.9 s, q90 2.6 s to 3.3 s, q53 1.5 s to 2.3 s, q99 about the same. Caveats: - This is one run, and short queries vary by 10-20% between identical runs, so the per-query differences under about 1 s may be noise. I have not repeated either side. - The 488 s run is the 1.2.0 release, and this PR is `main` at `a4d716b63` (75 commits after 1.2.0) plus 4 commits. So the comparison includes everything merged since 1.2.0, not only this PR. I have not run `main` at `a4d716b63` to separate them. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
