On Sat, Sep 1, 2012 at 7:21 PM, Brett Cannon <br...@python.org> wrote: > Now that I can run benchmarks against Python 2.7 and 3.3 simultaneously, I'm > ready to start updating the benchmarks. This involves two parts. > > One is moving benchmarks from PyPy over to the unladen repo on > hg.python.org/benchmarks. But I wanted to first make sure people don't view > the benchmarks as immutable (e.g. as Octane does: > https://developers.google.com/octane/faq). Since the benchmarks are always > relative between two interpreters their immutability isn't critical compared > to if we were to report some overall score. But it also means that any > changes made would throw off historical comparisons. For instance, if I take > PyPy's Mako benchmark (which does a lot more work), should it be named > mako_v2, or should we just replace mako wholesale? > > And the second is the same question for libraries. For instance, the unladen > benchmarks have Django 1.1a0 as the version which is rather ancient. And > with 1.5 coming out with provisional Python 3 support I obviously would like > to update it. But the same questions as with benchmarks crops up in > reference to immutability. Another thing is that 2to3 can't actually be > ported using 2to3 (http://bugs.python.org/issue15834) and so that itself > will require two versions -- a 2.x version (probably from Python 2.7's > stdlib) and a 3.x version (from the 3.2 stdlib) -- which already starts to > add interesting issues for me in terms of comparing performance (e.g. I will > have to probably update the 2.7 code to use io.BytesIO instead of > StringIO.StringIO to be on more equal footing). Similar thing goes for > html5lib which has developed its Python 3 support separately from its Python > 2 code. > > If we can't find a reasonable way to handle all of this then what I will do > is branch the unladen benchmarks for 2.x/3.x benchmarking, and then create > another branch of the benchmark suite to just be for Python 3.x so that we > can start fresh with a new set of benchmarks that will never change > themselves for benchmarking Python 3 itself. That would also mean we could > start of with whatever is needed from PyPy and unladen to have the optimal > benchmark runner for speed.python.org. > > _______________________________________________ > Speed mailing list > Speed@python.org > http://mail.python.org/mailman/listinfo/speed >
Ideally I would like benchmarks to be immutable (have _v is fine). However, updating libraries might not be immutable (after all, you're interested in *the speed of running django*), but maybe we should mark this somehow in the history so we don't compare apples to organges. Cheers, fijal _______________________________________________ Speed mailing list Speed@python.org http://mail.python.org/mailman/listinfo/speed
