I think this is the other paper I mentioned -- but it's behind a paywall and I don't have access. https://www.sciencedirect.com/science/article/abs/pii/S0167642323001508
On Sat, Jul 4, 2026 at 5:19 PM David Daly <[email protected]> wrote: > Hi all! I've been lurking. This all sounds great and thank you all for > the good work. I love seeing it. > > One more paper i would suggest is: > Creating a Virtuous Cycle in Performance Testing at MongoDB | Proceedings > of the ACM/SPEC International Conference on Performance Engineering > https://dl.acm.org/doi/10.1145/3427921.3450234 > > Which was a followup to the original change point paper. It made the point > that better detection led to more tests which led to more detection and the > need for better detection. > > There will also hopefully be a spec-rg paper coming out sometime in the > near future, rigorously evaluating data sets and change point detection > algorithms for regression detection. > > > Probably not worth including, but there were four papers submitted for a > related data challenge > https://icpe2022.spec.org/tracks-and-submissions/data-challenge-track > Including for completeness. See the digital library below and look for > the data challenge section. > > https://dl.acm.org/doi/proceedings/10.1145/3491204 > > One of those papers had a nice extension to a journal paper, but it will > take me a bit to find. That one got far enough that i wanted to take > things for to Mongo. > > On Tue, Jun 30, 2026, 11:31 AM Henrik Ingo <[email protected]> wrote: > >> On Mon, Jun 29, 2026 at 6:28 AM Alexander Sorokoumov < >> [email protected]> wrote: >> >> > Your plan also makes sense to me at a high-level. Regarding the blog, I >> do >> > wonder if we can borrow some of the articles from Nyrkio's blog that are >> > specifically about Apache Otava (e..g, [3]). >> >> >> Yes of course, perhaps also >> >> https://blog.nyrkio.com/2025/05/08/welcome-apache-otava-incubating-project/ >> which could be the first post on an otava blog as it is my look back on >> the >> history of the project until we became an ASF project. >> >> Note that for the Playground demo, perhaps the demo video is more useful >> than the blog post and one alternative is to repost the video but write a >> new blog post around it. >> >> >> >> > Also, there have been a bunch >> > of papers/talks about Otava/Hunter over the years. It would be great to >> > have a single page where all of them are linked. >> > >> > >> Yes! >> >> I seem to have unpublished a page where I used to keep such a list, but >> might as well collect one here. Or put these in a github issue: >> >> The original ICPE and DBTest papers: >> >> >> - >> >> Daly et.al. 2020: The Use of Change Point Detection to Identify >> Software >> Performance Regressions … >> <https://dl.acm.org/doi/10.1145/3358960.3375791> >> - >> >> Henrik Ingo & David Daly 2020: Automated system performance testing at >> MongoDB <https://dl.acm.org/doi/10.1145/3395032.3395323> >> - >> >> Fleming & Kolaczkowski et.al. 2023: Hunter: Using Change Point >> Detection >> to Hunt for Performance Regressions >> <https://dl.acm.org/doi/10.1145/3578244.3583719> >> >> >> Note: ACM tracks downloads, and is the official publisher of these anyway, >> so I try to link to those even if Google and others might list the >> corresponding Arxiv page first. >> >> This was the first use of change point detection outside of MongoDB after >> we had published the first ICPE paper: >> >> https://netflixtechblog.com/fixing-performance-regressions-before-they-happen-eab2602b86fe >> >> This presentation by Alex is very complete and IMO has been rather >> influential as well: >> >> https://www.confluent.io/events/kafka-summit-london-2024/automating-speed-a-proven-approach-to-preventing-performance-regressions-in/ >> >> >> >> Then there are some smaller publications, in smaller conferences or >> rejected papers. I'm sure I'm not aware of all of them, but I'm (co)author >> in >> 8 Years of Optimizing Apache Otava: How disconnected open source >> developers >> took an algorithm from n3 to constant time >> <https://arxiv.org/abs/2505.06758> >> Detection of Performance Changes in MooBench Results Using Nyrkiö on >> GitHub >> Actions <https://arxiv.org/abs/2510.11310>, >> >> henrik >> >> >> -- >> *nyrkio.com <http://nyrkio.com/>* ~ *Continuous Benchmarking as a >> Service* >> >> Henrik Ingo, CEO >> [email protected] LinkedIn: >> www.linkedin.com/in/heingo >> +358 40 569 7354 Twitter: >> twitter.com/h_ingo >> >
