On Thu, Apr 14, 2016 at 6:51 PM, Brett Cannon <br...@python.org> wrote: > > > On Wed, 13 Apr 2016 at 13:33 Zachary Ware <zachary.ware+py...@gmail.com> > wrote: >> >> On Wed, Apr 13, 2016 at 1:57 PM, Maciej Fijalkowski <fij...@gmail.com> >> wrote: >> > Hi >> > >> > I have a radical idea: to take a pypy benchmark suite, update the >> > libraries to newer ones and replace python benchmarks with that. The >> > main reason being that pypy has a much better coverage of things that >> > are not microbenchmarks, the list (in json): >> > >> > http://paste.pound-python.org/show/4YVq0fv6pv8rVOSmCTag/ >> > >> > Which is much more extensive than this: >> > >> > https://hg.python.org/benchmarks/file/tip/performance >> > >> > I'm willing to put *some* effort, what do people think? >> >> I'm in favor. My support has two conditions, though: >> >> 1) at least a majority of the benchmarks must be Python3 compatible. >> Preferably 2/3 compatible, but I assume all of the PyPy benchmarks are >> 2 compatible anyway. > > > Agreed (although I don't care about the 2/3 compatibility, just the 3 compat > ;) . > >> >> >> 2) and there should be an easy way to run the benchmarks against >> exactly 1 interpreter (for use with speed.python.org). I initially >> tried to set up speed.python.org using the PyPy benchmarks, but >> quickly ran into issues with trying to use 'nullpython.py' as the >> baseline Python. When I switched to using h.p.o/benchmarks, I added >> the '--raw' flag to perf.py which allows the benchmarks to be run on >> one interpreter instead of two. It was just a quick hack, though; I >> have no problems with that feature completely changing (even invoking >> it a different way is ok), so long as it exists. >> >> This project could probably start its life as >> github.com/python/benchmarks and save us from having to migrate >> h.p.o/benchmarks to GitHub. > > > Yep, I'm willing to postpone moving the benchmarks repo from hg.python.org > if works starts on this idea and then not move the old repo at all if this > succeeds. We could then make the people who care about the benchmarks the > maintainers of the new repository and contribute to it directly (which means > interested people from CPython, PyPy, Pyston, IronPython, and Jython). That > way we all get exposure to everyone's benchmarks and there's no more > benchmark fragmentation because some people disagree with another's approach > (i.e. no more benchmark silos among the Python implementations). > > And just because we're talking a new repo, would it be worth considering > relying on pip to grab libraries instead of embedding them in the > repository, hence shrinking the overall size of the repo? >
Both make sense - to benchmark against the latest lib X and to benchmark against a pinned lib X _______________________________________________ Speed mailing list Speed@python.org https://mail.python.org/mailman/listinfo/speed
