vito1317 commented on issue #5504: URL: https://github.com/apache/datafusion/issues/5504#issuecomment-4967191755
I've been working on the automation + analysis gaps discussed here, building on @alamb's [datafusion-benchmarking](https://github.com/alamb/datafusion-benchmarking) repo: [alamb/datafusion-benchmarking#16](https://github.com/alamb/datafusion-benchmarking/pull/16) adds: - **A nightly orchestrator** (cron/systemd; ownership-checked lockfile, per-step logs, `status.json` for monitoring, and effect assertions so a night that builds binaries but produces no results fails loudly instead of reporting green). It builds recent commits, runs the benchmarks, detects regressions, and regenerates the dashboard. - **Statistical regression detection** over the per-query median history, with two detectors: per-query significance thresholds learned from each query's own night-over-night noise (the IQR-fence approach rustc-perf uses), and E-divisive-means changepoint detection with a permutation significance test (the approach MongoDB published). Alerts are emitted as JSON/markdown, annotate the existing Plotly dashboard, and exit code 2 lets a cron wrapper page on regressions. As a sanity check against real data: running the detector over the existing `results_metal` history (383 per-commit revisions), it independently flags the `collect_statistics` default change (`2d7ae0926`) as a statistically significant changepoint on 7 queries — the same event that was previously hand-curated into `events.json` — without being told where to look. Everything is stdlib-only Python 3.9+ with ~100 unit tests. Happy to take this wherever it is most useful. In particular: should the analysis layer stay in the benchmarking repo, or would it be worth adapting the detectors to consume `dfbench` result JSONs directly — e.g. a `benchmarks/analyze.py` sibling of `compare.py` that reads N historical result files and flags statistically significant regressions instead of a pairwise mean comparison? Feedback on detection thresholds / alert format is very welcome. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
