On Tue, Feb 13, 2018 at 2:27 PM, Matti Picus <matti.pi...@gmail.com> wrote:
> I have begun to dive into the performance/perf code. My goal is to get pypy
> benchmarks running on http://speed.python.org. Since PyPy has a JIT, the
> benchmark runs must have a warmup stage. There are some first-cut warmup
> values hardcoded inside a few of the benchmarks. I think I would prefer to
> use a different mechanism - a separate calibrated data file alongside the
> performance benchmarks. We could start off with a rough guess for the
> benchmarks and get the system up and running, and then calibrate the warmups
> hopefully finding some statistical basis for the values.
Unfortunately, it is very difficult to determine when warmup has
occurred, even on a per-vm-per-benchmark basis. If you allow
benchmarks to run for long enough, some will fail to even reach a
steady state where warmup can be said to have finished.
We have a more detailed paper on this which might interest you:
and some software which can do something similar to what you propose
-- i.e. determine whether and when warmup has occurred:
this includes a script which can "diff" benchmarks between (say)
commits, or between different versions of a VM. There's an example
table on the link above, just hover over the table to see the diff.
If you want to have a go at using this, feel free to get in touch --
we'd be happy to help!
Dr. Sarah Mount, Research Associate, King's College London
Fellow of the Software Sustainability Institute
Speed mailing list