I think the action approach is sufficient for now. If we can get a dedicated GHA runner or similar, we can eventually move the benchmarks to a dedicated machine and still use the same API. -- Matt Sicker
> On Dec 28, 2021, at 13:39, Robert Middleton <[email protected]> wrote: > > I think adding it to github actions(while certainly not ideal) is at > least a step in the right direction. If/when we have dedicated > hardware to test it properly, we can then migrate it over. At least > having it setup to start with should make migration easier, plus even > if it's not super consistent we should at least be able to get a rough > order of magnitude over dozens of builds. > > -Robert Middleton > > On Tue, Dec 28, 2021 at 7:30 AM Volkan Yazıcı <[email protected]> wrote: >> >> Agreed with your remarks regarding the unreliability of benchmark results >> in the cloud. See my proposal in private@ to get some machines for >> continuous benchmarks. >> >> On Tue, Dec 28, 2021 at 10:17 AM Dominik Psenner <[email protected]> wrote: >> >>> Hi Stephen, >>> >>> The trouble with benchmarks in CI is that the numbers may be largely >>> unreliable, depending mostly on the hardware where it runs and in general >>> the surrounding environment. Chances are high that the benchmarks will not >>> produce comparable results. >>> >>> It would however be good to provide some tools to run the (same) benchmarks >>> manually. >>> >>> When run on the same hardware with different codebases or on different >>> hardware with the same codebase, the outcome may provide interesting and >>> comparable insights. >>> >>> Warm regards >>> -- >>> Sent from my phone. Typos are a kind gift to anyone who happens to find >>> them. >>> >>> On Tue, Dec 28, 2021, 07:46 Stephen Webb <[email protected]> wrote: >>> >>>> Hi, >>>> >>>> Robert has created a benchmark that I thought would be nice to integrate >>>> into CI. >>>> >>>> I see the Log4J has some benchmarks actions which are currently run >>>> manually with results posted to github pages. >>>> >>>> Do you consider this a useful/optimal approach? >>>> >>>> Would an threshold which an action could check for each PR be useful? >>>> >>>> Regards >>>> Stephen Webb >>>> >>>> < >>>> >>> https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail >>>>> >>>> Virus-free. >>>> www.avast.com >>>> < >>>> >>> https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail >>>>> >>>> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> >>>> >>>
