On Sat, Sep 1, 2012 at 9:45 PM, Gregory P. Smith <g...@krypto.org> wrote: > > > On Sat, Sep 1, 2012 at 10:21 AM, Brett Cannon <br...@python.org> wrote: >> >> Now that I can run benchmarks against Python 2.7 and 3.3 simultaneously, >> I'm ready to start updating the benchmarks. This involves two parts. >> >> One is moving benchmarks from PyPy over to the unladen repo on >> hg.python.org/benchmarks. But I wanted to first make sure people don't view >> the benchmarks as immutable (e.g. as Octane does: >> https://developers.google.com/octane/faq). Since the benchmarks are always >> relative between two interpreters their immutability isn't critical compared >> to if we were to report some overall score. But it also means that any >> changes made would throw off historical comparisons. For instance, if I take >> PyPy's Mako benchmark (which does a lot more work), should it be named >> mako_v2, or should we just replace mako wholesale? > > > I dislike benchmark immutability. The rest of the world including your > local computing environment where benchmarks run continues to change around > benchmarks which really makes using historical benchmark data from a run on > an old version for comparison to a recent modern run pointless.
So far we (pypy) managed to maintain enough of the environment under control to have meaningful historical data. We have the same machine and we monitor whether changes introduce something new or not. Of course ideally, it's impossible, but for real world what we're doing is good enough. Cheers, fijal _______________________________________________ Speed mailing list Speed@python.org http://mail.python.org/mailman/listinfo/speed
