I think adding it to github actions(while certainly not ideal) is at least a step in the right direction. If/when we have dedicated hardware to test it properly, we can then migrate it over. At least having it setup to start with should make migration easier, plus even if it's not super consistent we should at least be able to get a rough order of magnitude over dozens of builds.
-Robert Middleton On Tue, Dec 28, 2021 at 7:30 AM Volkan Yazıcı <vol...@yazi.ci> wrote: > > Agreed with your remarks regarding the unreliability of benchmark results > in the cloud. See my proposal in private@ to get some machines for > continuous benchmarks. > > On Tue, Dec 28, 2021 at 10:17 AM Dominik Psenner <dpsen...@gmail.com> wrote: > > > Hi Stephen, > > > > The trouble with benchmarks in CI is that the numbers may be largely > > unreliable, depending mostly on the hardware where it runs and in general > > the surrounding environment. Chances are high that the benchmarks will not > > produce comparable results. > > > > It would however be good to provide some tools to run the (same) benchmarks > > manually. > > > > When run on the same hardware with different codebases or on different > > hardware with the same codebase, the outcome may provide interesting and > > comparable insights. > > > > Warm regards > > -- > > Sent from my phone. Typos are a kind gift to anyone who happens to find > > them. > > > > On Tue, Dec 28, 2021, 07:46 Stephen Webb <swebb2...@gmail.com> wrote: > > > > > Hi, > > > > > > Robert has created a benchmark that I thought would be nice to integrate > > > into CI. > > > > > > I see the Log4J has some benchmarks actions which are currently run > > > manually with results posted to github pages. > > > > > > Do you consider this a useful/optimal approach? > > > > > > Would an threshold which an action could check for each PR be useful? > > > > > > Regards > > > Stephen Webb > > > > > > < > > > > > https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail > > > > > > > Virus-free. > > > www.avast.com > > > < > > > > > https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail > > > > > > > <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> > > > > >
