> On 27 Sep 2021, at 19:02, Andrew MacLeod <[email protected]> wrote:
>
> On 9/27/21 11:39 AM, Maxim Kuvyrkov via Gcc wrote:
>>> On 27 Sep 2021, at 16:52, Aldy Hernandez <[email protected]> wrote:
>>>
>>> [CCing Jeff and list for broader audience]
>>>
>>> On 9/27/21 2:53 PM, Maxim Kuvyrkov wrote:
>>>> Hi Aldy,
>>>> Your patch seems to slow down 471.omnetpp by 8% at -O3. Could you please
>>>> take a look if this is something that could be easily fixed?
>>> First of all, thanks for chasing this down. It's incredibly useful to have
>>> these types of bug reports.
>> Thanks, Aldy, this is music to my ears :-).
>>
>> We have built this automated benchmarking CI that bisects code-speed and
>> code-size regressions down to a single commit. It is still
>> work-in-progress, and I’m forwarding these reports to patch authors, whose
>> patches caused regressions. If GCC community finds these useful, we can
>> also setup posting to one of GCC’s mailing lists.
>
> I second that this sort of thing is incredibly useful. I don't suppose its
> easy to do the reverse?... let patch authors know when they've caused a
> significant improvement? :-) That would be much less common I suspect, so
> perhaps not worth it :-)
We do this occasionally, when identifying a regression in a patch revert commit
:-). Seriously, though, it’s an easy enough code-change to the metric, but we
are maxing out our benchmarking capacity with current configuration matrix.
>
> Its certainly very useful when we are making a wholesale change to a pass
> which we think is beneficial, but aren't sure.
>
> And a followup question... Sometimes we have no good way of determining the
> widespread run-time effects of a change. You seem to be running SPEC/other
> things continuously then?
We continuously run SPEC CPU2006 on {arm,aarch64}-{-Os/-O2/-O3}-{no LTO/LTO}
matrix for GNU and LLVM toolchains.
In the GNU toolchain we track master branches and latest-release branches of
Binutils, GCC and Glibc — and detect code-speed and code-size regressions
across all toolchain components.
> Does it run like once a day/some-time-period, and if you note a regression,
> narrow it down?
Configurations that track master branches have 3-day intervals. Configurations
that track release branches — 6 days. If a regression is detected it is
narrowed down to component first — binutils, gcc or glibc — and then the commit
range of the component is bisected down to a specific commit. All. Done.
Automatically.
I will make a presentation on this CI at the next GNU Tools Cauldron.
> Regardless, I think it could be very useful to be able to see the results of
> anything you do run at whatever frequency it happens.
Thanks!
--
Maxim Kuvyrkov
https://www.linaro.org
