Hi all! I've been lurking. This all sounds great and thank you all for the good work. I love seeing it.
One more paper i would suggest is: Creating a Virtuous Cycle in Performance Testing at MongoDB | Proceedings of the ACM/SPEC International Conference on Performance Engineering https://dl.acm.org/doi/10.1145/3427921.3450234 Which was a followup to the original change point paper. It made the point that better detection led to more tests which led to more detection and the need for better detection. There will also hopefully be a spec-rg paper coming out sometime in the near future, rigorously evaluating data sets and change point detection algorithms for regression detection. Probably not worth including, but there were four papers submitted for a related data challenge https://icpe2022.spec.org/tracks-and-submissions/data-challenge-track Including for completeness. See the digital library below and look for the data challenge section. https://dl.acm.org/doi/proceedings/10.1145/3491204 One of those papers had a nice extension to a journal paper, but it will take me a bit to find. That one got far enough that i wanted to take things for to Mongo. On Tue, Jun 30, 2026, 11:31 AM Henrik Ingo <[email protected]> wrote: > On Mon, Jun 29, 2026 at 6:28 AM Alexander Sorokoumov < > [email protected]> wrote: > > > Your plan also makes sense to me at a high-level. Regarding the blog, I > do > > wonder if we can borrow some of the articles from Nyrkio's blog that are > > specifically about Apache Otava (e..g, [3]). > > > Yes of course, perhaps also > https://blog.nyrkio.com/2025/05/08/welcome-apache-otava-incubating-project/ > which could be the first post on an otava blog as it is my look back on the > history of the project until we became an ASF project. > > Note that for the Playground demo, perhaps the demo video is more useful > than the blog post and one alternative is to repost the video but write a > new blog post around it. > > > > > Also, there have been a bunch > > of papers/talks about Otava/Hunter over the years. It would be great to > > have a single page where all of them are linked. > > > > > Yes! > > I seem to have unpublished a page where I used to keep such a list, but > might as well collect one here. Or put these in a github issue: > > The original ICPE and DBTest papers: > > > - > > Daly et.al. 2020: The Use of Change Point Detection to Identify > Software > Performance Regressions … > <https://dl.acm.org/doi/10.1145/3358960.3375791> > - > > Henrik Ingo & David Daly 2020: Automated system performance testing at > MongoDB <https://dl.acm.org/doi/10.1145/3395032.3395323> > - > > Fleming & Kolaczkowski et.al. 2023: Hunter: Using Change Point > Detection > to Hunt for Performance Regressions > <https://dl.acm.org/doi/10.1145/3578244.3583719> > > > Note: ACM tracks downloads, and is the official publisher of these anyway, > so I try to link to those even if Google and others might list the > corresponding Arxiv page first. > > This was the first use of change point detection outside of MongoDB after > we had published the first ICPE paper: > > https://netflixtechblog.com/fixing-performance-regressions-before-they-happen-eab2602b86fe > > This presentation by Alex is very complete and IMO has been rather > influential as well: > > https://www.confluent.io/events/kafka-summit-london-2024/automating-speed-a-proven-approach-to-preventing-performance-regressions-in/ > > > > Then there are some smaller publications, in smaller conferences or > rejected papers. I'm sure I'm not aware of all of them, but I'm (co)author > in > 8 Years of Optimizing Apache Otava: How disconnected open source developers > took an algorithm from n3 to constant time > <https://arxiv.org/abs/2505.06758> > Detection of Performance Changes in MooBench Results Using Nyrkiö on GitHub > Actions <https://arxiv.org/abs/2510.11310>, > > henrik > > > -- > *nyrkio.com <http://nyrkio.com/>* ~ *Continuous Benchmarking as a Service* > > Henrik Ingo, CEO > [email protected] LinkedIn: > www.linkedin.com/in/heingo > +358 40 569 7354 Twitter: > twitter.com/h_ingo >
