alamb opened a new issue, #5504: URL: https://github.com/apache/arrow-datafusion/issues/5504
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.** As we make changes to DataFusion, some changes impact performance and some do not. Right now we mostly rely on reviewers to judge when a change could make an impact on performance, and if so run the appropriate benchmarks. This means that 1. We may miss some performance regressions (such as https://github.com/apache/arrow-datafusion/issues/5325) 2. Since the benchmarks are not run regularly it is hard to know how to interpret results, and some seem to have bitrotted over time 3. The wide variety of available benchmarks (e.g. https://github.com/apache/arrow-datafusion/issues/5502) makes it hard to know which ones to run and how to determine if performance has improved or regressed for particular changes **Describe the solution you'd like** I would like 1. A system that runs DataFusion benchmarks regularly on main 2. Some automated way to see if a particular PR has improved or regressed performance 3. Bonus: a webpage that shows performance over time. Databend has a great example https://perf.databend.rs/ ** Suggestion ** I believe conbench, https://conbench.ursa.dev/, which is partially integrated into the repo already, is intended for exactly this usecase. Using conbench would be nice as it appears to be actively maintained and has resources and is already hosted The integration is https://github.com/apache/arrow-datafusion/tree/main/conbench and was added in https://github.com/apache/arrow-datafusion/pull/1791 by @dianaclarke You can see its integration as it posts comments on PRs after merge such as https://github.com/apache/arrow-datafusion/pull/5476#issuecomment-1456748444 **Describe alternatives you've considered** We could also figure out a new system as well **Additional context** <!-- Add any other context or screenshots about the feature request here. --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
