On 6/10/20 6:00 PM, Andres Freund wrote: > On June 10, 2020 2:13:51 PM PDT, David Rowley <dgrowle...@gmail.com> wrote: >>On Thu, 11 Jun 2020 at 02:13, Tom Lane <t...@sss.pgh.pa.us> wrote: >>> I have in the past scraped the latter results and tried to make sense >>of >>> them. They are *mighty* noisy, even when considering just one animal >>> that I know to be running on a machine with little else to do. >> >>Do you recall if you looked at the parallel results or the serially >>executed ones? >> >>I imagine that the parallel ones will have much more noise since we >>run the tests up to 20 at a time. I imagine probably none, or at most >>not many of the animals have enough CPU cores not to be context >>switching a lot during those the parallel runs. I thought the serial >>ones would be better but didn't have an idea of they'd be good enough >>to be useful. > > I'd assume that a rolling average (maybe 10 runs or so) would hide noise > enough to see at least some trends even for parallel runs. > > We should be able to prototype this with a few queries over the bf database, > right?
This seems to me like a perfect use case for control charts: https://en.wikipedia.org/wiki/Control_chart They are designed specifically to detect systematic changes in an environment with random noise. Joe -- Crunchy Data - http://crunchydata.com PostgreSQL Support for Secure Enterprises Consulting, Training, & Open Source Development
