vito1317 commented on issue #5504: URL: https://github.com/apache/datafusion/issues/5504#issuecomment-4962527462
I took a stab at the remaining piece of this issue — running benchmarks regularly and tracking history with automated regression detection — building directly on @alamb's datafusion-benchmarking repo: alamb/datafusion-benchmarking#16 It adds a cron-first nightly orchestrator (no CI-vendor dependency, per the guidance earlier in this thread) plus a statistical analysis layer addressing the "we'll probably just make a bunch of data we never look at" concern: per-query noise-adaptive thresholds (rustc-perf style IQR fences) and E-divisive changepoint detection (MongoDB's approach), both stdlib-only Python. Detected changes are annotated on the existing dashboard automatically. As a sanity check, the detector independently re-discovers the `collect_statistics` change (2d7ae09) as a changepoint on 7 ClickBench queries when run over the historical per-commit dataset. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
