Hi, Would it be possible to run a new instance of CodeSpeed (the website behing speed.python.org) which would run the "performance" benchmark suite rather than the "benchmarks" benchmark suite? And would it be possible to run it on CPython (2.7 and 3.5 branches) and PyPy (master branch, maybe also the py3k branch)?
I found https://github.com/tobami/codespeed/ but I didn't look at it right now. I guess that some code should be written to convert perf JSON file to the format expected by CodeSpeed? FYI I released performance 0.2 yesterday. JSON files now contain the version of the benchmark suite ("performance_version: 0.2"). I plan to use semantic version: increase the major version (ex: upgrade to 0.3, but later it will be 1.x, 2.x, etc.) when benchmark results are considered to not be compatible. For example, I upgraded Django (from 1.9 to 1.10) and Chameleon (from 2.22 to 2.24) in performance 0.2. The question is how to upgrade the performance to a new major version: should we drop previous benchmark results? Maybe we should put the performance version in the URL, and use "/latest/" by default. Only /latest/ would get new results, and /latest/ would restart from an empty set of results when performance is upgraded? Another option, less exciting, is to never upgrade benchmarks. The benchmarks project *added* new benchmarks when a dependency was "upgraded". In fact, the old dependency was kept and a new dependency (full copy of the code in fact ;-)) was added. So it has django, django_v2, django_v3, etc. The problem is that it still uses Mercurial 1.2 which was released 7 years ago (2009)... Since it's painful to upgrade, most dependencies were outdated. Do you care of old benchmark results? It's quite easy to regenerate them (on demand?) if needed, no? Using Mercurial and Git, it's easy to update to any old revisions to run again a benchmark on an old version of CPython / PyPy / etc. Victor _______________________________________________ Speed mailing list Speed@python.org https://mail.python.org/mailman/listinfo/speed
