On Tue, Feb 13, 2018 at 2:27 PM, Matti Picus <matti.pi...@gmail.com> wrote: > > I have begun to dive into the performance/perf code. My goal is to get pypy > benchmarks running on http://speed.python.org. Since PyPy has a JIT, the > benchmark runs must have a warmup stage. There are some first-cut warmup > values hardcoded inside a few of the benchmarks. I think I would prefer to > use a different mechanism - a separate calibrated data file alongside the > performance benchmarks. We could start off with a rough guess for the > benchmarks and get the system up and running, and then calibrate the warmups > hopefully finding some statistical basis for the values. >
Unfortunately, it is very difficult to determine when warmup has occurred, even on a per-vm-per-benchmark basis. If you allow benchmarks to run for long enough, some will fail to even reach a steady state where warmup can be said to have finished. We have a more detailed paper on this which might interest you: http://soft-dev.org/pubs/html/barrett_bolz-tereick_killick_mount_tratt__virtual_machine_warmup_blows_hot_and_cold_v6/ and some software which can do something similar to what you propose -- i.e. determine whether and when warmup has occurred: http://soft-dev.org/src/warmup_stats/ this includes a script which can "diff" benchmarks between (say) commits, or between different versions of a VM. There's an example table on the link above, just hover over the table to see the diff. If you want to have a go at using this, feel free to get in touch -- we'd be happy to help! Regards, Sarah -- Dr. Sarah Mount, Research Associate, King's College London Fellow of the Software Sustainability Institute twitter: @snim2 _______________________________________________ Speed mailing list Speed@python.org https://mail.python.org/mailman/listinfo/speed
