andygrove opened a new issue, #1123: URL: https://github.com/apache/datafusion-comet/issues/1123
### What is the problem the feature request solves? I have been comparing Comet's performance with [Ballista](https://github.com/apache/datafusion-ballista#performance) because I believe that performance should be similar. I have been focusing on TPC-H q3 for comparison. In my local environment, Ballista is currently ~3x faster than Comet (Comet is ~20 seconds and Ballista is ~6 seconds). Comet and Ballista are executing nearly identical plans (I am testing Comet with the `comet-parquet-exec branch`, which uses DataFusion's ParquetExec). The main difference between Comet and Ballista is that Comet shuffle uses Arrow FFI to pass the output from a native query into the JVM and then uses Arrow FFI to pass those batches back into native code with a new native plan that performs the shuffle write. The following chart shows a summary of the native execution time for this query. ScanExec accounts for 66% of the native execution time.  Hypothetically, if the ScanExec time were reduced to zero, then the native execution time would be 3x faster, matching Ballista. This epic is to track issues related to improving shuffle performance. Issues: - [ ] https://github.com/apache/datafusion-comet/issues/1115 - [ ] https://github.com/apache/datafusion-comet/issues/977 ### Describe the potential solution _No response_ ### Additional context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
