Hi Victor Despite the fact that I was not able to find time to run your stuff yet, thanks for all the awesome work!
On Thu, Oct 20, 2016 at 12:56 PM, Victor Stinner <victor.stin...@gmail.com> wrote: > Hi, > > Last months, I worked a lot on benchmarks. I ran benchmarks, analyzed > results in depth (up to the hardware and kernel drivers!), I wrote new > tools and enhanced existing tools. > > * I wrote a new perf module which runs benchmarks in a reliable way > and contains a LOT of features: collect metadata, JSON file format, > commands to compare, render an histogram, etc. > > * I rewrote the Python benchmark suite: the old benchmarks Mercurial > repository moved to a new performance GitHub project which uses my > perf module and contains more benchmarks. > > * I also made minor enhancements to timeit in Python 3.7 -- some dev > don't want major changes to not "break the backward compatibility". > > For timeit, I suggest to use my perf tool which includes a reliable > timeit command and has much more features like --duplicate (repeat the > statements to reduce the cost of the outer loop) and --compare-to > (compare two versions of Python), but also all builtin perf features > (JSON output, statistics, histogram, etc.). > > I added benchmarks from PyPy and Pyston benchmark suites to > performance: performance 0.3.1 contains 51 benchmark scripts which run > a total of 121 benchmarks. Example of tested Python modules: > > * SQLAlchemy > * Dulwich (full Git implementation in Python) > * Mercurial (currently only the startup time) > * html5lib > * pyaes (AES crypto cipher in pure Python) > * sympy > * Tornado (HTTP client and server) > * Django (sadly, only the template engine right now, Pyston contains > HTTP benchmarks) > * pathlib > * spambayes > > More benchmarks will be added later. It would be nice to add > benchmarks on numpy for example, numpy is important for a large part > of our community. > > All these (new or updated) tools can now be used to take smarter > decisions on optimizations. Please don't push any optimization anymore > without providing reliable benchmark results! > > > My first major action was to close the latest attempt to > micro-optimize int+int in Python/ceval.c, > http://bugs.python.org/issue21955 : I closed the issue as rejected, > because there is no significant speedup on benchmarks other than two > (tiny) microbenchmarks. To make sure that no one looses its time on > trying to micro-optimize int+int, I even added a comment to > Python/ceval.c :-) > > https://hg.python.org/cpython/rev/61fcb12a9873 > "Please don't try to micro-optimize int+int" > > > The perf and performance are now well tested: Travis CI runs tests on > the new commits and pull requests, and the "tox" command can be used > locally to test different Python versions, pep8, doc, ... in a single > command. > > > Next steps: > > * Run performance 0.3.1 on speed.python.org: the benchmark runner is > currently stopped (and still uses the old benchmarks project). The > website part may be updated to allow to download full JSON files which > includes *all* information (all timings, metadata and more). > > * I plan to run performance on CPython 2.7, CPython 3.7, PyPy and PyPy > 3. Maybe also CPython 3.5 and CPython 3.6 if they don't take too much > resources. > > * Later, we can consider adding more implementations of Python: > Jython, IronPython, MicroPython, Pyston, Pyjion, etc. All benchmarks > should be run on the same hardware to be comparable. > > * Later, we might also allow other projects to upload their own > benchmark results, but we should find a solution to groups benchmark > results per benchmark runner (ex: at least by the hostname, perf JSON > contains the hostname) to not compare two results from two different > hardware > > * We should continue to add more benchmarks to the performance > benchmark suite, especially benchmarks more representative of real > applications (we have enough microbenchmarks!) > > > Links: > > * perf: http://perf.readthedocs.io/ > * performance: https://github.com/python/performance > * Python Speed mailing list: https://mail.python.org/mailman/listinfo/speed > * https://speed.python.org/ (currently outdated, and don't use performance > yet) > > See https://pypi.python.org/pypi/performance which contains even more > links to Python benchmarks (PyPy, Pyston, Numba, Pythran, etc.) > > Victor > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/fijall%40gmail.com _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
