gruuya commented on issue #5504: URL: https://github.com/apache/arrow-datafusion/issues/5504#issuecomment-1973862429
Ok, I think I'm up to speed with the history of this issue now. To me it seems that there are 2 separate action items discussed here: 1. CI benchmark that will comment on PRs once merged to main, and possibly on open PRs after invoking some hook (points 1 and 2 in the issue description). This will be just a summary of the perf comparison between the base commit and the current/merge commit. This might be done using github shared runners, as presumably the variance wouldn't be too wild during the same run. 2. As an extension of 1. above, a way to persist this info, and ideally visualize it over time. This would involve a bit more effort, as it would require setting up (and maintaining) dedicated infra to control for the variance. I guess the idea was to be checking in the [benchmark numbers](https://github.com/apache/arrow-datafusion/issues/6107) into some(this?) github repo, as opposed to a cloud DB somewhere? Also looks like both [conbench](https://github.com/apache/arrow/pull/39592#issuecomment-1969889622) and [bencher](https://github.com/bencherdev/bencher?tab=readme-ov-file#example-pr-comment) are eligible to handle the above. IMO it's best to try and tackle 1. for now, as this would present a big win, and would pave the way to 2 as well. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
