On Tue, Feb 13, 2018 at 2:27 PM, Matti Picus <matti.pi...@gmail.com> wrote:
> I have begun to dive into the performance/perf code. My goal is to get pypy 
> benchmarks running on http://speed.python.org. Since PyPy has a JIT, the 
> benchmark runs must have a warmup stage. There are some first-cut warmup 
> values hardcoded inside a few of the benchmarks. I think I would prefer to 
> use a different mechanism - a separate calibrated data file alongside the 
> performance benchmarks. We could start off with a rough guess for the 
> benchmarks and get the system up and running, and then calibrate the warmups 
> hopefully finding some statistical basis for the values.

Unfortunately, it is very difficult to determine when warmup has
occurred, even on a per-vm-per-benchmark basis. If you allow
benchmarks to run for long enough, some will fail to even reach a
steady state where warmup can be said to have finished.

We have a more detailed paper on this which might interest you:


and some software which can do something similar to what you propose
-- i.e. determine whether and when warmup has occurred:


this includes a script which can "diff" benchmarks between (say)
commits, or between different versions of a VM. There's an example
table on the link above, just hover over the table to see the diff.

If you want to have a go at using this, feel free to get in touch --
we'd be happy to help!



Dr. Sarah Mount, Research Associate, King's College London
Fellow of the Software Sustainability Institute
twitter: @snim2
Speed mailing list

Reply via email to