Anyone tracking  the homepage of speed.pypy.org will notice a drop in
the pypy-vs-cpython ratio from ~4.1x to ~3.5x. TL;DR: I am redoing the
cpython runs and the results changed. They will change a bit more
going forward.

Part of the motivation to move to benchmarker2 was to udate all the
things and get gcc14 into the builds. This worked and the recent
releases of pypy8 on linux use gcc14. But... I wrote some scripting to
run the benchmarker2 buildbot since I want it to run inside a vm,
pinning to cpus dedicated only to benchmarking. This is the
https://github.com/pypy/buildbot/blob/main/start-benchmarker2.sh
script. It had a typo: the PATH addition was not properly escaped. All
the benchmarks run since benchmarker2 started have used the system gcc
8.5 rather than the /opt/rh/gcc-toolset-14 gcc14.

I noticed this when trying to properly run cpython 3.11 and 3.12 with
both our legacy benchmarks and the pyperformance ones. Rather than the
manual benchmark runs I have been doing in the past, I added a
buildbot runner for CPython. This exposed a further typo in the RPATH
escaping I was using, so the enable-shared builds of CPython may not
have been using the libpython I wanted.

Where we stand now? I reran the 3.11 and 3.12 CPython benchmarks with
the RPATH fix and replaced the old results. This moved the needle on
the pypy vs cpython comparisons
and dropped PyPy from 4.1x to 3.5x CPython, where it was on the older
benchmarker machine.

But that was all with gcc8.5. So now there is a new environment
'benchmarker2-gcc14' (hidden under the "historical" accordion on the
comparison, timeline, changes pages). This uses gcc14. Eventually,
when we have enough results to get some kind of a timeline, I will
flip the names, benchmarker2 will become benchmarker2-gcc8.5 and
benchmarker2-gcc14 will become benchmarker2. I think the bottom line
will stay somewhere around 3.5x for legacy benchmarks.

The whole point of this exercise is to slowly move to pyperformance
benchmark comparisons.

Sorry for the mess.
Matti
_______________________________________________
pypy-dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3//lists/pypy-dev.python.org
Member address: [email protected]

Reply via email to