Author: Matti Picus <[email protected]> Branch: nikola Changeset: r966:6f20654cdb2b Date: 2019-12-29 08:55 +0200 http://bitbucket.org/pypy/pypy.org/changeset/6f20654cdb2b/
Log: convert the source/*.txt pages diff --git a/pages/compat.rst b/pages/compat.rst new file mode 100644 --- /dev/null +++ b/pages/compat.rst @@ -0,0 +1,154 @@ +.. title: Python compatibility +.. slug: compat +.. date: 2019-12-28 16:14:02 UTC +.. tags: +.. category: +.. link: +.. description: + +PyPy implements the Python language version 2.7.13. It supports all of the core +language, passing Python test suite (with minor modifications that were +already accepted in the main python in newer versions). It supports most +of the commonly used Python `standard library modules`_; details below. + +PyPy3 implements the Python language version 3.6.9. It has been released, +but Python is a large language and it is quite possible that a few things are missing. + +.. class:: download_menu + + `List of installable top 1000 PyPI packages`_ + +PyPy has support for the `CPython C API`_, however there are constructs +that are `not compatible`. We strongly advise use of `CFFI`_ +instead. CFFI come builtin with PyPy. Many libraries will require +a bit of effort to work, but there are known success stories. Check out +PyPy blog for updates, as well as the `Compatibility Wiki`__. + +.. __: https://bitbucket.org/pypy/compatibility/wiki/Home + +C extensions need to be recompiled for PyPy in order to work. Depending on +your build system, it might work out of the box or will be slightly harder. + +Standard library modules supported by PyPy. Note that large parts of python +library are implemented in pure python, so they don't have to be listed +there. Please just check if it imports. If it imports, it should work. + +* ``__builtin__, __pypy__, _ast, _cffi_backend, _codecs, _collections, _continuation, _csv, _file, _hashlib, _io, _locale, _lsprof, _md5, _minimal_curses, _multibytecodec, _multiprocessing, _numpypy, _pickle_support, _pypyjson, _random, _rawffi, _sha, _socket, _sre, _ssl, _struct, _testing, _warnings, _weakref, array, binascii, bz2, cStringIO, cmath, cppyy, cpyext, crypt, errno, exceptions, fcntl, gc, imp, itertools, marshal, math, mmap, operator, parser, posix, pwd, pyexpat, pypyjit, select, signal, symbol, sys, termios, thread, time, token, unicodedata, zipimport, zlib`` + +Supported, but written in pure Python: + +* ``cPickle, ctypes, datetime, dbm, _functools, grp, readline, resource, sqlite3, syslog`` + +All modules that are pure python in CPython of course work. + +Python libraries known to work under PyPy (the list is not exhaustive). +A `fuller list`_ is available. + +* ctypes + +* django + +* sqlalchemy + +* flask + +* twisted + +* pylons + +* divmod's nevow + +* pyglet + +* Pillow (the PIL fork) + +* `lxml`_ + +* NumPy + +The main difference that is not going to be fixed is that PyPy does +not support refcounting semantics. The following code won't fill the +file immediately, but only after a certain period of time, when the GC +does a collection: + +.. code-block:: python + + open("filename", "w").write("stuff") + +The proper fix is + +.. code-block:: python + + f = open("filename", "w") + f.write("stuff") + f.close() + +or using the ``with`` keyword + +.. code-block:: python + + with open("filename", "w") as f: + f.write("stuff") + +The same problem---not closing your files---can also show up if your +program opens a large number of files without closing them explicitly. +In that case, you can easily hit the system limit on the number of file +descriptors that are allowed to be opened at the same time. + +Since release 5.4, PyPy can be run with the command-line option ``-X +track-resources`` (as in, ``pypy -X track-resources myprogram.py``). +This produces a ResourceWarning when the GC closes a non-closed file or +socket. The traceback for the place where the file or socket was +allocated is given as well, which aids finding places where ``close()`` +is missing. + +Similarly, remember that you must ``close()`` a non-exhausted +generator in order to have its pending ``finally`` or ``with`` +clauses executed immediately: + +.. code-block:: python + + def mygen(): + with foo: + yield 42 + + for x in mygen(): + if x == 42: + break # foo.__exit__ is not run immediately! + + # fixed version: + gen = mygen() + try: + for x in gen: + if x == 42: + break + finally: + gen.close() + +More generally, ``__del__()`` methods are not executed as predictively +as on CPython: they run "some time later" in PyPy (or not at all if +the program finishes running in the meantime). See `more details +here`_. + +Note that PyPy returns unused memory to the operating system if there +is a madvise() system call (at least Linux, OS X, BSD) or on Windows. It is +important to realize that you may not see this in ``top``. The unused +pages are marked with ``MADV_FREE``, which tells the system "if you +need more memory at some point, grab this page". As long as memory is +plentiful, the ``RES`` column in ``top`` might remains high. (Exceptions to +this rule are systems with no ``MADV_FREE``, where we use +``MADV_DONTNEED``, which forcefully lowers the ``RES``. This includes +Linux <= 4.4.) + +A more complete list of known differences is available at `our dev site`_. + +.. _`CPython C API`: http://docs.python.org/c-api/ +.. _`CFFI`: http://cffi.readthedocs.org/ +.. _`not compatible`: http://doc.pypy.org/en/latest/cpython_differences.html#c-api-differences +.. _`standard library modules`: http://docs.python.org/library/ +.. _`our dev site`: http://pypy.readthedocs.org/en/latest/cpython_differences.html +.. _`more details here`: http://pypy.readthedocs.org/en/latest/cpython_differences.html#differences-related-to-garbage-collection-strategies +.. _`compatibility wiki`: https://bitbucket.org/pypy/compatibility/wiki/Home +.. _`lxml`: https://github.com/amauryfa/lxml/tree/cffi/ +.. _`List of installable top 1000 PyPI packages`: http://packages.pypy.org +.. _`fuller list`: http://packages.pypy.org diff --git a/pages/contact.rst b/pages/contact.rst new file mode 100644 --- /dev/null +++ b/pages/contact.rst @@ -0,0 +1,25 @@ +.. title: Contact +.. slug: contact +.. date: 2019-12-28 16:14:02 UTC +.. tags: +.. category: +.. link: +.. description: + + +* irc: **#pypy** on **irc.freenode.net** + +* mailing list: `pypy-dev at python.org`__ + +* for security related issues, non-public funding enquiries etc. please contact [email protected] + +* the bitbucket `bug tracker`_ (registration required to open new issues or to comment) + +* more on our `dev site`_. + +* code on `bitbucket`_. + +.. __: http://mail.python.org/mailman/listinfo/pypy-dev +.. _`bug tracker`: https://bitbucket.org/pypy/pypy/issues?status=new&status=open +.. _`dev site`: http://doc.pypy.org +.. _`bitbucket`: https://bitbucket.org/pypy/pypy/overview diff --git a/pages/download.rst b/pages/download.rst new file mode 100644 --- /dev/null +++ b/pages/download.rst @@ -0,0 +1,512 @@ +.. title: Download and Install +.. slug: download +.. date: 2019-12-28 16:14:02 UTC +.. tags: +.. category: +.. link: +.. description: + +.. class:: download_menu + + There are `nightly binary builds`_ available. Those builds are not always + as stable as the release, but they contain numerous bugfixes and + performance improvements. + +We provide binaries for x86, aarch64, ppc64 and s390x running on different operating systems such as +Linux, Mac OS X and Windows (`what's new in PyPy 7.3.0?`_): + +* the Python2.7 compatible release — **PyPy2.7 v7.3.0** + +* the Python3.6 compatible release — **PyPy3.6 v7.3.0** + +* the Python2.7 Software Transactional Memory special release — **PyPy-STM 2.5.1** (Linux x86-64 only) + +.. _what's new in PyPy 7.3.0?: http://doc.pypy.org/en/latest/release-v7.3.0.html + + +.. class:: download_menu + + * Download + + * `Default (with a JIT Compiler)`_ + * `Other versions`_ + + * `Installing`_ (optional) + * `Installing more modules`_ + * `Building from source`_ + * `Packaging`_ + * `Checksums`_ + +.. _`Default (with a JIT Compiler)`: + +"JIT Compiler" version +------------------------------- + +These binaries include a Just-in-Time compiler. They only work on +x86 CPUs that have the SSE2_ instruction set (most of +them do, nowadays), or on x86-64 CPUs. They also contain `stackless`_ +extensions, like `greenlets`_. + +Linux binaries and common distributions +--------------------------------------- + +Since version 7.3, the linux x86 binaries in the links below ship with versions +of OpenSSL, SQLite3, libffi, expat, and TCL/TK binary libraries linked in. This +make the binaries "portable" so that they should run on any current glibc-based +linux platform. The ideas were adopted from the `portable-pypy`_ package. + +This solution to the portability problem means that the versions of the +packaged libraries are frozen to the version shipped, so updating your system +libraries will not affect this installation of PyPy. Also see the note about +SSL certificates below. + +For aarch64, s390x, and ppc64, the binaries target a specific operating system. +These binaries are dynamically linked, and thus might not be usable due to the +sad story of linux binary compatibility. This means that **Linux binaries are +only usable on the distributions written next to them** unless you're ready to +hack your system by adding symlinks to the libraries it tries to open. There +are better solutions: + +* download PyPy from your release vendor (usually an outdated + version): `Ubuntu`_ (`PPA`_), `Debian`_, `Homebrew`_, MacPorts, + `Fedora`_, `Gentoo`_ and `Arch`_ are known to package PyPy, with various + degrees of being up-to-date. + +* use ``sudo snap install --classic <package>``, where ``<package>`` is + ``pypy`` or `pypy3``. Snap is a non-vendor specific package manager for + linux, and repackages the download tarballs below with the latest platform- + specific libraries (again, without changing ``libffi``). + +* `recompile the CFFI-based`_ TCL/TK, OpenSSL, or sqlite3 modules, using system + libraries and the scripts in ``pypy/lib_pypy``. This solution will not solve + compatibility issues with libffi, since that is baked into PyPy. + +* or translate_ your own PyPy. + +.. class:: download_menu + + SSL Certificates + + While the linux binaries ship an OpenSSL library, they do not ship a + certificate store for SSL certificates. If you wish to use SSL module, + you will need a valid certificate store. You can use the `certifi`_ package + and set ``SSL_CERT_FILE`` to ``certifi.where()`` or install your platform + certificates which should be discovered by the ``_ssl`` module. + + +.. _`Ubuntu`: http://packages.ubuntu.com/search?keywords=pypy&searchon=names +.. _`PPA`: https://launchpad.net/~pypy/+archive/ppa +.. _`Debian`: http://packages.debian.org/sid/pypy +.. _`Fedora`: http://fedoraproject.org/wiki/Features/PyPyStack +.. _`Gentoo`: http://packages.gentoo.org/package/dev-python/pypy +.. _`Homebrew`: https://github.com/Homebrew/homebrew-core/blob/master/Formula/pypy.rb +.. _`Arch`: https://wiki.archlinux.org/index.php/PyPy +.. _`portable-pypy`: https://github.com/squeaky-pl/portable-pypy#portable-pypy-distribution-for-linux +.. _`recompile the CFFI-based`: https://doc.pypy.org/en/latest/build.html#build-cffi-import-libraries-for-the-stdlib +.. _`certifi`: https://pypi.org/project/certifi/ + +.. _release: + +Python2.7 compatible PyPy 7.3.0 +------------------------------- + +.. class:: download_menu + +* `Linux x86 binary (32bit, built on CenOS6)`__ +* `Linux x86-64 binary (64bit, built on CentOS6)`__ +* `Mac OS X binary (64bit)`__ +* FreeBSD x86 and x86_64: see FreshPorts_ +* `Windows binary (32bit)`__ (you might need the VC runtime library + installer `vcredist.x86.exe`_.) +* `Linux aarch64 binary (64bit, built on Ubuntu 18.04)`__ (see ``[1]`` below) +* `PowerPC PPC64 Linux binary (64bit big-endian, Fedora 20)`__ (see ``[1]`` below) +* `PowerPC PPC64le Linux binary (64bit little-endian, Fedora 21)`__ (see ``[1]`` below) +* `s390x Linux binary (built on Redhat Linux 7.2)`__ (see ``[1]`` below) +* `Source (tar.bz2)`__; `Source (zip)`__. See below for more about the sources. +* `All our downloads,`__ including previous versions. We also have a + mirror_, but please use only if you have troubles accessing the links above + + +.. __: https://bitbucket.org/pypy/pypy/downloads/pypy2.7-v7.3.0-linux32.tar.bz2 +.. __: https://bitbucket.org/pypy/pypy/downloads/pypy2.7-v7.3.0-linux64.tar.bz2 +.. __: https://bitbucket.org/pypy/pypy/downloads/pypy2.7-v7.3.0-osx64.tar.bz2 +.. __: https://bitbucket.org/pypy/pypy/downloads/pypy2.7-v7.3.0-win32.zip +.. __: https://bitbucket.org/pypy/pypy/downloads/pypy2.7-v7.3.0-aarch64.tar.bz2 +.. __: https://bitbucket.org/pypy/pypy/downloads/pypy2.7-v7.3.0-ppc64.tar.bz2 +.. __: https://bitbucket.org/pypy/pypy/downloads/pypy2.7-v7.3.0-ppc64le.tar.bz2 +.. __: https://bitbucket.org/pypy/pypy/downloads/pypy2.7-v7.3.0-s390x.tar.bz2 +.. __: https://bitbucket.org/pypy/pypy/downloads/pypy2.7-v7.3.0-src.tar.bz2 +.. __: https://bitbucket.org/pypy/pypy/downloads/pypy2.7-v7.3.0-src.zip +.. _`vcredist.x86.exe`: https://www.microsoft.com/en-us/download/details.aspx?id=52685 +.. __: https://bitbucket.org/pypy/pypy/downloads +.. _mirror: http://buildbot.pypy.org/mirror/ +.. _FreshPorts: http://www.freshports.org/lang/pypy + + +Python 3.6 compatible PyPy3.6 v7.3.0 +------------------------------------ + +.. class:: download_menu + +* `Linux x86-64 binary (64bit, built on CentOS6)`__ +* `Linux x86 binary (32bit, built on CentOS6)`__ +* `Mac OS X binary (64bit)`__ (High Sierra >= 10.13, not for Sierra and below) +* `Windows binary (32bit)`__ (you might need the VC runtime library + installer `vcredist.x86.exe`_.) +* `Linux aarch64 binary (64bit, built on Ubuntu 18.04)`__ (see ``[1]`` below) +* `PowerPC PPC64 Linux binary (64bit big-endian, Fedora 20)`__ (see ``[1]`` below) +* `PowerPC PPC64le Linux binary (64bit little-endian, Fedora 21)`__ (see ``[1]`` below) +* `s390x Linux binary (built on Redhat Linux 7.2)`__ (see ``[1]`` below) +* `Source (tar.bz2)`__; `Source (zip)`__. See below for more about the sources. +* `All our downloads,`__ including previous versions. We also have a + mirror_, but please use only if you have troubles accessing the links above + +.. __: https://bitbucket.org/pypy/pypy/downloads/pypy3.6-v7.3.0-linux64.tar.bz2 +.. __: https://bitbucket.org/pypy/pypy/downloads/pypy3.6-v7.3.0-linux32.tar.bz2 +.. __: https://bitbucket.org/pypy/pypy/downloads/pypy3.6-v7.3.0-osx64.tar.bz2 +.. __: https://bitbucket.org/pypy/pypy/downloads/pypy3.6-v7.3.0-win32.zip +.. __: https://bitbucket.org/pypy/pypy/downloads/pypy3.6-v7.3.0-aarch64.tar.bz2 +.. __: https://bitbucket.org/pypy/pypy/downloads/pypy3.6-v7.3.0-ppc64.tar.bz2 +.. __: https://bitbucket.org/pypy/pypy/downloads/pypy3.6-v7.3.0-ppc64le.tar.bz2 +.. __: https://bitbucket.org/pypy/pypy/downloads/pypy3.6-v7.3.0-s390x.tar.bz2 +.. __: https://bitbucket.org/pypy/pypy/downloads/pypy3.6-v7.3.0-src.tar.bz2 +.. __: https://bitbucket.org/pypy/pypy/downloads/pypy3.6-v7.3.0-src.zip +.. __: https://bitbucket.org/pypy/pypy/downloads + + +If your CPU is really, really old, it may be a x86-32 without SSE2. +There is untested support for manually translating PyPy's JIT without +SSE2 (``--jit-backend=x86-without-sse2``) but note that your machine +is probably low-spec enough that running CPython on it is a better +idea in the first place. + +``[1]:`` stating it again: the Linux binaries are provided for the +distributions listed here. **If your distribution is not exactly this +one, it won't work,** you will probably see: ``pypy: error while loading shared +libraries: ...``. + +PyPy-STM 2.5.1 +------------------------------ + +This is a special version of PyPy! See the `Software Transactional +Memory`_ (STM) documentation. + +* `PyPy-STM Linux x86-64 binary (64bit, tar.bz2 built on Ubuntu 12.04 - 16.04)`__ + +.. _`Software Transactional Memory`: http://doc.pypy.org/en/latest/stm.html +.. __: https://bitbucket.org/pypy/pypy/downloads/pypy-stm-2.5.1-linux64.tar.bz2 + + +.. _`Other versions (without a JIT)`: + +Other versions +------------------------------- + +The other versions of PyPy are: + +* The most up-to-date `nightly binary builds`_ with a JIT, if the official + release is too old for what you want to do. There are versions for + different libc on this site too. + +* Reverse debugger: This version enables debugging your Python + programs by going forward and backward in time. See the `RevDB + documentation`__. + +.. __: https://bitbucket.org/pypy/revdb/ + +* Old-style sandboxing: A special safe version. + *This is NOT the version announced in-development during 2019!* + Read the docs about sandboxing_. + This version is **not supported** and not actively maintained. You + will likely have to fix some issues yourself, or checkout an old + version, or otherwise play around on your own. We provide this + documentation only for historical reasons. Please do not use in + production. For reference, there are some very old, unmaintained + binaries for Linux (32bit__, 64bit__). + +.. __: https://bitbucket.org/pypy/pypy/downloads/pypy-1.8-sandbox-linux64.tar.bz2 +.. __: https://bitbucket.org/pypy/pypy/downloads/pypy-1.8-sandbox-linux.tar.bz2 +.. _`sandbox docs`: http://doc.pypy.org/en/latest/sandbox.html + +.. _`nightly binary builds`: http://buildbot.pypy.org/nightly/trunk/ + +Installing +---------- + +All binary versions are packaged in a ``tar.bz2`` or ``zip`` file. When +uncompressed, they run in-place. You can uncompress them +either somewhere in your home directory or, say, in ``/opt``. +If you want, put a symlink from somewhere like +``/usr/local/bin/pypy`` to ``/path/to/pypy_expanded/bin/pypy``. Do +not move or copy the executable ``pypy`` outside the tree --- put +a symlink to it, otherwise it will not find its libraries. + + +Installing more modules +------------------------------- + +There are as yet few distribution-ready packages. +We recommend installing ``pip``, which is the standard package +manager of Python. It works like it does on CPython as explained in the +`installation documentation`_. + +If you use your distribution's PyPy package we recommend you install packages +into a virtualenv. If you try to build a module and the build process complains +about "missing Python.h", you may need to install the pypy-dev package. + +.. _installation documentation: http://doc.pypy.org/en/latest/install.html + +.. _translate: + +Building from source +-------------------- + +(see more build instructions_) + + +1. Get the source code. The preferred way is to checkout the current + trunk using Mercurial_. The trunk usually works and is of course + more up-to-date. The following command should run in about 7 minutes + nowadays if you have hg >= 3.7 (it is much slower with older versions): + + .. code-block:: bash + + hg clone https://bitbucket.org/pypy/pypy + + The trunk contains PyPy 2. For PyPy 3, switch to the correct branch: + + .. code-block:: bash + + # switch to the branch that implements Python 3.6 + hg update py3.6 + + Alternatively, get one of the following smaller packages for the source at + the same revision as the above binaries: + + * `pypy2.7-v7.3.0-src.tar.bz2`__ (sources, PyPy 2 only) + * `pypy3.6-v7.3.0-src.tar.bz2`__ (sources, PyPy 3 only) + + .. __: https://bitbucket.org/pypy/pypy/downloads/pypy2.7-v7.3.0-src.tar.bz2 + .. __: https://bitbucket.org/pypy/pypy/downloads/pypy3.6-v7.3.0-src.tar.bz2 + + +2. Make sure you **installed the dependencies.** See the list here__. + + .. __: http://pypy.readthedocs.org/en/latest/build.html#install-build-time-dependencies + +3. Enter the ``goal`` directory: + + .. code-block:: bash + + cd pypy/pypy/goal + +4. Run the ``rpython`` script. Here are the common combinations + of options (works also with ``python`` instead of ``pypy``; + requires CPython 2.7 or PyPy 2, even to build PyPy 3): + + .. code-block:: bash + + # get the JIT version + pypy ../../rpython/bin/rpython -Ojit targetpypystandalone + # get the no-jit version + pypy ../../rpython/bin/rpython -O2 targetpypystandalone + # get the sandbox version + pypy ../../rpython/bin/rpython -O2 --sandbox targetpypystandalone + +5. Enjoy Mandelbrot ``:-)`` It takes on the order of half an hour to + finish the translation, and about 3GB of RAM on a 32-bit system + and about 5GB on 64-bit systems. (Do not start a translation on a + machine with insufficient RAM! It will just swap forever. See + notes below in that case.) + +6. If you want to install this PyPy as root, please read the next section, + Packaging_. + +Notes: + +* It is recommended to use PyPy to do translations, instead of using CPython, + because it is twice as fast. You should just start by downloading an + official release of PyPy (with the JIT). If you really have to use CPython + then note that we are talking about CPython 2.7 here, not CPython 3.x. + (Older versions like 2.6 are out.) + +* On some 32-bit systems, the address space limit of 2 or 3 GB of RAM + can be an issue. More generally you may be just a little bit low of + RAM. First note that 2 GB is really not enough nowadays; on Windows + you first need to refer to the `Windows build instructions`_. More + precisely, translation on 32-bit takes at this point 2.7 GB if PyPy is + used and 2.9 GB if CPython is used. There are two workarounds: + + 1. use PyPy, not CPython. If you don't have any PyPy so far, not even + an older version, then you need to build one first, with some parts + removed. So, first translate with ``...rpython -Ojit + targetpypystandalone --withoutmod-micronumpy --withoutmod-cpyext``, + then copy ``pypy-c`` and ``libpypy_c.so`` somewhere else, and finally + call it with ``...pypy-c ../../rpython/bin/rpython -Ojit``. + + 2. if even using PyPy instead of CPython is not enough, try to tweak + some internal parameters. Example (slower but saves around 400MB): + + .. code-block:: bash + + PYPY_DONT_RUN_SUBPROCESS=1 PYPY_GC_MAX_DELTA=200MB \ + pypy --jit loop_longevity=300 ../../rpython/bin/rpython -Ojit --source + # then read the next point about --source + +* You can run translations with ``--source``, which only builds the C + source files (and prints at the end where). Then you can ``cd`` there + and execute ``make``. This is another way to reduce memory usage. + Note that afterwards, you have to run manually ``pypy-c + .../pypy/tool/build_cffi_imports.py`` if you want to be able to import + the cffi-based modules. + +* Like other JITs, PyPy doesn't work out of the box on some Linux + distributions that trade full POSIX compliance for extra security + features. E.g. with PAX, you have to run PyPy with ``paxctl -cm``. + This also applies to translation (unless you use CPython to run the + translation and you specify ``--source``). + +.. _instructions: http://pypy.readthedocs.org/en/latest/build.html +.. _`x86 (IA-32)`: http://en.wikipedia.org/wiki/IA-32 +.. _`x86-64`: http://en.wikipedia.org/wiki/X86-64 +.. _SSE2: http://en.wikipedia.org/wiki/SSE2 +.. _`contact us`: contact.html +.. _`sandboxing`: features.html#sandboxing +.. _`stackless`: http://www.stackless.com/ +.. _`greenlets`: http://pypy.readthedocs.org/en/latest/stackless.html#greenlets +.. _`Windows build instructions`: http://doc.pypy.org/en/latest/windows.html#preparing-windows-for-the-large-build +.. _`shadow stack`: http://pypy.readthedocs.org/en/latest/config/translation.gcrootfinder.html +.. _Mercurial: https://www.mercurial-scm.org/ + +Packaging +--------- + +Once PyPy is translated from source the binary package similar to those +provided in the section `Default (with a JIT Compiler)`_ above could be +easily created with ``package.py`` script: + +.. code-block:: bash + + cd ./pypy/pypy/tool/release/ + python package.py --help #for information + python package.py --archive-name pypy-my-own-package-name + +It is recommended to use package.py because custom scripts will +invariably become out-of-date. If you want to write custom scripts +anyway, note an easy-to-miss point: some modules are written with CFFI, +and require some compilation. If you install PyPy as root without +pre-compiling them, normal users will get errors: + +* PyPy 2.5.1 or earlier: normal users would see permission errors. + Installers need to run ``pypy -c "import gdbm"`` and other similar + commands at install time; the exact list is in `package.py`_. Users + seeing a broken installation of PyPy can fix it after-the-fact if they + have sudo rights, by running once e.g. ``sudo pypy -c "import gdbm``. + +* PyPy 2.6 and later: anyone would get ``ImportError: no module named + _gdbm_cffi``. Installers need to run ``pypy _gdbm_build.py`` in the + ``lib_pypy`` directory during the installation process (plus others; + see the exact list in `package.py`_). Users seeing a broken + installation of PyPy can fix it after-the-fact, by running ``pypy + /path/to/lib_pypy/_gdbm_build.py``. This command produces a file + called ``_gdbm_cffi.pypy-41.so`` locally, which is a C extension + module for PyPy. You can move it at any place where modules are + normally found: e.g. in your project's main directory, or in a + directory that you add to the env var ``PYTHONPATH``. + +.. _`package.py`: https://bitbucket.org/pypy/pypy/src/default/pypy/tool/release/package.py + +Checksums +--------- + +Here are the checksums for each of the downloads of PyPy 7.3.0, 7.2.0, 7.1.1, 7.1.0 + +pypy2.7-7.3.0 sha256:: + + a3dd8d5e2a656849fa344dce4679d854a19bc4a096a0cf62b46a1be127a5d56c pypy2.7-v7.3.0-aarch64.tar.bz2 + eac1308b7d523003a5f6d20f58406d52ab14611bcec750122ae513a5a35110db pypy2.7-v7.3.0-linux32.tar.bz2 + f4950a54378ac637da2a6defa52d6ffed96af12fcd5d74e1182fb834883c9826 pypy2.7-v7.3.0-linux64.tar.bz2 + ca7b056b243a6221ad04fa7fc8696e36a2fb858396999dcaa31dbbae53c54474 pypy2.7-v7.3.0-osx64.tar.bz2 + d254b82a00021339762198e41ba7f72316010d0f9bd4dcd7b0755185da9c005e pypy2.7-v7.3.0-s390x.tar.bz2 + b0b25c7f8938ab0fedd8dedf26b9e73c490913b002b484c1b2f19d5844a518de pypy2.7-v7.3.0-src.tar.bz2 + 42dc84a277e7a5e635fe39bbd745f06135902c229a257123332b7555800d915b pypy2.7-v7.3.0-src.zip + a9e3c5c983edba0313a41d3c1ab55b080816c4129e67a6c272c53b9dbcdd97ec pypy2.7-v7.3.0-win32.zip + +pypy3.6-7.3.0 sha256:: + + b900241bca7152254c107a632767f49edede99ca6360b9a064141267b47ef598 pypy3.6-v7.3.0-aarch64.tar.bz2 + 7045b295d38ba0b5ee65bd3f078ca249fcf1de73fedeaab2d6ad78de2eab0f0e pypy3.6-v7.3.0-linux32.tar.bz2 + d3d549e8f43de820ac3385b698b83fa59b4d7dd6cf3fe34c115f731e26ad8856 pypy3.6-v7.3.0-linux64.tar.bz2 + 87b2545dad75fe3027b4b2108aceb9fdadcdd24e61ae312ac48b449fdd452bf3 pypy3.6-v7.3.0-osx64.tar.bz2 + 0fe2f7bbf42ea88b40954d7de773a43179a44f40656f2f58201524be70699544 pypy3.6-v7.3.0-s390x.tar.bz2 + 48d12c15fbcbcf4a32882a883195e1f922997cde78e7a16d4342b9b521eefcfa pypy3.6-v7.3.0-src.tar.bz2 + 8ae9efd0a2aadb19e892bbd07eca8ef51536296a3ef93964149aceba511e79ca pypy3.6-v7.3.0-src.zip + 30e6870c4f3d8ef91890a6556a98080758000ba7c207cccdd86a8f5d358998c1 pypy3.6-v7.3.0-win32.zip + +pypy2.7-7.2.0 sha256:: + + 57b0be053c6a5f069e23b843f38863cf7920f5eef7bc89f2e086e5c3a28a2ba9 pypy2.7-v7.2.0-aarch64.tar.bz2 + 76d666e5aee54b519d6ec1af4ef0cbdc85f7f9276dd554e97deb026adfd0c936 pypy2.7-v7.2.0-linux32.tar.bz2 + 05acf28e6a243026ecad933b9361d8f74b41f00818071b76b38c4694cc4c9599 pypy2.7-v7.2.0-linux64.tar.bz2 + 36aa2f2440e762333569118dd0b3d5371d575c40966effa194d116c5453ddb52 pypy2.7-v7.2.0-osx64.tar.bz2 + fb51150a4ce94b0ca8587899ba69c41fc58a6b35c5340ea6926376ecb9cfcac4 pypy2.7-v7.2.0-ppc64.tar.bz2 + 5c4224525657c29b815cb2c6b3f9bc5a267368cc6adf0fedb235a6052929f65f pypy2.7-v7.2.0-ppc64le.tar.bz2 + bb7ae585ecb4d904c890e28a2c5b6bd379f57cc3d9e38ff45597ff54fa935eaa pypy2.7-v7.2.0-s390x.tar.bz2 + 55cb7757784fbe3952102447f65b27d80e6c885a464a7af1a9ce264492439dcc pypy2.7-v7.2.0-src.tar.bz2 + 897038550614d558f9f6718409b107e27903ef2b2b57ec250939d1b1ebdf0aba pypy2.7-v7.2.0-src.zip + 956eeaaaac053e5d0917e77a3d2ad1933ab5561eb3e6e71235780b5aa5fd2bb7 pypy2.7-v7.2.0-win32.zip + +pypy2.7-7.1.1 sha256:: + + 41ca390a76ca0d47b8353a0d6a20d5aab5fad8b0bb647b960d8c33e873d18ef5 pypy2.7-v7.1.1-linux32.tar.bz2 + 73b09ef0860eb9ad7997af3030b22909806a273d90786d78420926df53279d66 pypy2.7-v7.1.1-linux64.tar.bz2 + 31a17294dec96c2191885c776b4ee02112957dc874f7ba03e570537a77b78c35 pypy2.7-v7.1.1-osx64.tar.bz2 + 1ef94c3a9c67c2335cee0b21753036b4696ed588b9d54b7b8036a6ae47f7001d pypy2.7-v7.1.1-s390x.tar.bz2 + 5f06bede6d71dce8dfbfe797aab26c8e35cb990e16b826914652dc093ad74451 pypy2.7-v7.1.1-src.tar.bz2 + d9b07a2954ad6dbde94feffd848311e2b5169563d33e3e9f17969579b01a4158 pypy2.7-v7.1.1-src.zip + 9c59226311f216a181e70ee7b5aa4d9665a15d00f24ae02acec9af7d96355f63 pypy2.7-v7.1.1-win32.zip + +pypy2.7-7.1.0 sha256:: + + 44ec91e8cb01caab289d8763c203f3aaf288d14325a6c42692bd1ac4e870d758 pypy2.7-v7.1.0-linux32.tar.bz2 + fef176a29a2ef068c00c8098e59dab935ca6e956f089672b3f7351da95a034f5 pypy2.7-v7.1.0-linux64.tar.bz2 + 8be43685ce718b0768387450fc6dc395d60809b778b6146c353ef67826022153 pypy2.7-v7.1.0-osx64.tar.bz2 + b065f55741bcb37863f1eca30ce91c9d79159371a6994100930cdc2ede3237bc pypy2.7-v7.1.0-s390x.tar.bz2 + b051a71ea5b4fa27d0a744b28e6054661adfce8904dcc82500716b5edff5ce4b pypy2.7-v7.1.0-src.tar.bz2 + e60ce30f9947844da43daaa7658adc0c05330681305225954114772f42df06ec pypy2.7-v7.1.0-src.zip + 76658c9ad679d562b8b6a09d006caa666406337b9834ff56db16980c5e549f20 pypy2.7-v7.1.0-win32.zip + +pypy3.6-7.2.0 sha256:: + + f82dc9dc6c692417ee9727f23beae75364a5757ebdc657a2a1d0010ac3ad17ab pypy3.6-v7.2.0-aarch64.tar.bz2 + 45e99de197cb3e974cfc8d45e0076ad2066852e61e56b3eafd1237efafd2c43e pypy3.6-v7.2.0-linux32.tar.bz2 + aa128e555ad0fe5c4c15104ae0903052bd232b6e3a73f5fe023d27b8fd0d6089 pypy3.6-v7.2.0-linux64.tar.bz2 + 836abb0ec303b90a684533711ed3b8269d3e8c64805b595e410920abdea678ac pypy3.6-v7.2.0-osx64.tar.bz2 + 14021d196e393b3a6d2395ab94ceec347753715e37223efe4c50b7c141b351a2 pypy3.6-v7.2.0-ppc64.tar.bz2 + 6aef73a3b68e9a6c062cadd83d3db16790960cf97401ca6f2aad2195e9b05c35 pypy3.6-v7.2.0-ppc64le.tar.bz2 + a11da8118064db102d159e9221319c428b298c4a87f26166fd6ae94be8d6ae0d pypy3.6-v7.2.0-s390x.tar.bz2 + 0d7c707df5041f1593fe82f29c40056c21e4d6cb66554bbd66769bd80bcbfafc pypy3.6-v7.2.0-src.tar.bz2 + 405ac35695dd374d5ea192cb44cb47231f9a65812cc7b6549df33df12ffe54db pypy3.6-v7.2.0-src.zip + c926f622bec24a8b348591d631717ace83b3a6c3c2dac02b157b622b97d1fc9c pypy3.6-v7.2.0-win32.zip + +pypy3.6-7.1.1 sha256:: + + cb11ef4b0df569c28390b1ee93029159e1b90bfbad98df6abd629d5203b2abd9 pypy3.6-v7.1.1-linux32.tar.bz2 + 8014f63b1a34b155548852c7bf73aab2d41ebddf2c8fb603dc9dd8509be93db0 pypy3.6-v7.1.1-linux64.tar.bz2 + a5c2f2bfa2b4a4d29e8a67baab95699b169054066df218a14f171bb84a6df0c0 pypy3.6-v7.1.1-osx64.tar.bz2 + 4a91bf2d9a142b6dbf82b5301cb510535ae9a54e1645546b2e0735a7b5ed85ba pypy3.6-v7.1.1-s390x.tar.bz2 + 6a3ef876e3691a54f4cff045028ec3be94ab9beb2e99f051b83175302c1899a8 pypy3.6-v7.1.1-src.tar.bz2 + 4a3ebeb767740f2dc0b886d02797d21d7d69f154cf951bb991c19bd485e6cae1 pypy3.6-v7.1.1-src.zip + 8b513b254de5f31890f5956569de9aec3a0a91d7aba72fc89d66901f4a8ccf49 pypy3.6-v7.1.1-win32.zip + +pypy 3.6-v7.1.0 sha256:: + + + 031bfac61210a6e161bace0691b854dc15d01b0e624dc0588c544ee5e1621a83 pypy3.6-v7.1.0-linux32.tar.bz2 + 270dd06633cf03337e6f815d7235e790e90dabba6f4b6345c9745121006925fc pypy3.6-v7.1.0-linux64.tar.bz2 + d46e005ba095cb4a7006079ffbf4fe63c18cf5e9d8ce9ce8383efc1a4863ab5b pypy3.6-v7.1.0-osx64.tar.bz2 + 243cd0cc188a94c1f064f402ae72b8ba4303eb3137eac53c53826472b8005098 pypy3.6-v7.1.0-s390x.tar.bz2 + faa81f469bb2a7cbd22c64f22d4b4ddc5a1f7c798d43b7919b629b932f9b1c6f pypy3.6-v7.1.0-src.tar.bz2 + 4858e7e8a0007bc3b381bd392208b28d30889a4e5a88a3c28e3d9dc4f25b654e pypy3.6-v7.1.0-src.zip + 77a0576a3d518210467f0df2d0d9a1892c664566dc02f25d974c2dbc6b4749e7 pypy3.6-v7.1.0-win32.zip + diff --git a/pages/features.rst b/pages/features.rst new file mode 100644 --- /dev/null +++ b/pages/features.rst @@ -0,0 +1,168 @@ +.. title: What is PyPy? +.. slug: features +.. date: 2019-12-28 16:14:02 UTC +.. tags: +.. category: +.. link: +.. description: + +What is PyPy? +=========================================================== + +PyPy is a replacement for CPython. It is built using the RPython +language that was co-developed with it. The main reason to use it +instead of CPython is speed: it runs generally faster (see next section). + +**PyPy** implements **Python 2.7.13 and 3.6.9**. +It supports all of the core language, passing the Python 2.7 test suite +and most of the 3.6 test suite (with minor modifications) It supports most of +the commonly used Python standard library modules. For known differences with +CPython, see our `compatibility`_ page. + +The following CPU architectures are supported and maintained: + +* `x86 (IA-32)`_ and `x86_64`_ +* `ARM`_ platforms (ARMv6 or ARMv7, with VFPv3) +* `AArch64`_ +* `PowerPC`_ 64bit both little and big endian +* `System Z (s390x)`_ + +PyPy's x86 version runs on several operating systems, such as Linux +(32/64 bits), Mac OS X (64 bits), Windows (32 bits), OpenBSD, FreeBSD. +All non-x86 versions are only supported on Linux. + +If you are interested in helping to move forward, see our `howtohelp`_ page. + +.. _`compatibility`: compat.html +.. _`x86 (IA-32)`: http://en.wikipedia.org/wiki/IA-32 +.. _`x86_64`: http://en.wikipedia.org/wiki/X86_64 +.. _`ARM`: http://en.wikipedia.org/wiki/ARM +.. _`AArch64`: http://en.wikipedia.org/wiki/AArch64 +.. _`PowerPC`: https://de.wikipedia.org/wiki/PowerPC +.. _`System Z (s390x)`: https://de.wikipedia.org/wiki/System/390 +.. _`howtohelp`: howtohelp.html + + + +The main features of PyPy: +-------------------------- + +Speed +----- + +Our `main executable`_ comes with a Just-in-Time compiler. It is +`really fast`_ in running most benchmarks --- including very large and +complicated Python applications, not just 10-liners. + +There are two cases that you should be aware where PyPy will *not* be +able to speed up your code: + +* Short-running processes: if it doesn't run for at least a few seconds, + then the JIT compiler won't have enough time to warm up. + +* If all the time is spent in run-time libraries (i.e. in C functions), + and not actually running Python code, the JIT compiler will not help. + +So the case where PyPy works best is when executing long-running +programs where a significant fraction of the time is spent executing +Python code. This is the case covered by the majority of `our +benchmarks`_, but not all of them --- the goal of PyPy is to get speed +but still support (ideally) any Python program. + +.. _`main executable`: download.html#with-a-jit-compiler +.. _`really fast`: http://speed.pypy.org/ +.. _`our benchmarks`: http://speed.pypy.org/ + + +Memory usage +-------------------------- + +Memory-hungry Python programs (several hundreds of MBs or more) might +end up taking less space than they do in CPython. It is not always +the case, though, as it depends on a lot of details. Also note that +the baseline is higher than CPython's. + + +Stackless +-------------------------- + +Support for Stackless_ and greenlets are now integrated in the normal +PyPy. More detailed information is available here__. + +.. _Stackless: http://www.stackless.com/ +.. __: http://doc.pypy.org/en/latest/stackless.html + + +Other features +--------------------------------------- + +PyPy has many secondary features and semi-independent +projects. We will mention here: + +* **Other languages:** we also implemented other languages that makes + use of our RPython toolchain: Prolog_ (almost complete), as + well as Smalltalk_, JavaScript_, Io_, Scheme_ and Gameboy_. + + There is also a Ruby implementation called Topaz_ and a PHP implementation + called HippyVM_. + + +Sandboxing +-------------------- + +PyPy's *sandboxing* is a working prototype for the idea of running untrusted +user programs. Unlike other sandboxing approaches for Python, PyPy's does not +try to limit language features considered "unsafe". Instead we replace all +calls to external libraries (C or platform) with a stub that communicates +with an external process handling the policy. + ++-----------------------------------------------------------------------------+ +| **Please be aware that it is a prototype only.** *It needs work to become | +| more complete, and you are welcome to help. In particular, almost none | +| of the extension modules work (not even* ``time`` *), and* ``pypy_interact``| +| *is merely a demo. Also, a more complete system would include a way | +| to do the same as* ``pypy_interact`` *from other languages than Python, | +| to embed a sandboxed interpreter inside programs written in other | +| languages.* | ++-----------------------------------------------------------------------------+ + +To run the sandboxed process, you need to get the full sources and +build ``pypy-sandbox`` from it (see `Building from source`_). These +instructions give you a ``pypy-c`` that you should rename to +``pypy-sandbox`` to avoid future confusion. Then run:: + + cd pypy/sandbox + pypy_interact.py path/to/pypy-sandbox + # don't confuse it with pypy/goal/pyinteractive.py! + +You get a fully sandboxed interpreter, in its own filesystem hierarchy +(try ``os.listdir('/')``). For example, you would run an untrusted +script as follows:: + + mkdir virtualtmp + cp untrusted.py virtualtmp/ + pypy_interact.py --tmp=virtualtmp pypy-sandbox /tmp/untrusted.py + +Note that the path ``/tmp/untrusted.py`` is a path inside the sandboxed +filesystem. You don't have to put ``untrusted.py`` in the real ``/tmp`` +directory at all. + +To read more about its features, try ``pypy_interact.py --help`` or go to +`our documentation site`_. + +.. _`Building from source`: download.html#building-from-source +.. _`our documentation site`: http://pypy.readthedocs.org/en/latest/sandbox.html + + + + +.. _`the cli-jit branch`: https://bitbucket.org/pypy/pypy/src/cli-jit +.. _`contact us`: contact.html +.. _Prolog: https://bitbucket.org/cfbolz/pyrolog/ +.. _Smalltalk: https://bitbucket.org/pypy/lang-smalltalk/ +.. _JavaScript: https://bitbucket.org/pypy/lang-js/ +.. _Io: https://bitbucket.org/pypy/lang-io/ +.. _Scheme: https://bitbucket.org/pypy/lang-scheme/ +.. _Gameboy: https://bitbucket.org/pypy/lang-gameboy/ +.. _Topaz: http://topazruby.com/ +.. _HippyVM: http://www.hippyvm.com/ diff --git a/pages/index.rst b/pages/index.rst --- a/pages/index.rst +++ b/pages/index.rst @@ -1,4 +1,4 @@ -.. title: index +.. title: Welcome to PyPy .. slug: index .. date: 2019-12-28 16:14:02 UTC .. tags: diff --git a/pages/people.rst b/pages/people.rst new file mode 100644 --- /dev/null +++ b/pages/people.rst @@ -0,0 +1,154 @@ +.. title: The PyPy Team (from 2008) +.. slug: people +.. date: 2019-12-28 16:14:02 UTC +.. tags: +.. category: +.. link: +.. description: + + +Armin Rigo +========== + +.. image:: images/people/arigo.png + +Armin Rigo is a former researcher at the Heinrich-Heine Universitat +Düsseldorf (Germany). He studied Mathematics at the University +of Lausanne (Switzerland), obtained his Ph.D. in Logic and Set +Theory at the Free University of Brussels (Belgium) in 2002, and +worked at the University of Southampton (UK) until 2005. He is +the author of Psyco, the first just-in-time compiler for Python. +He is one of the founders and lead developers of the PyPy project +which began in 2003. He has taken part in all areas, from the Python +language definition to the RPython translation framework, +including the garbage collector and the tracing just-in-time +compiler. + +Maciej Fijałkowski +================== + +.. image:: images/people/fijal_thumb.png + +Maciej is a freelancer working mostly on PyPy for the past several years. +He's a core developer since 2006, working on all kinds of parts in +the entire codebase including JIT, GC and assembler backends. +Maciej has been going to many conferences, advertising PyPy to a broader +audience for the past several years, including a keynote at Pycon 2010. +He's also the main maintainer of +`jitviewer`_, a tool for analyzing performance of your python programs under +PyPy. + +.. _`jitviewer`: https://bitbucket.org/pypy/jitviewer + +Carl Friedrich Bolz +=================== + +.. image:: images/people/cfbolz.jpg + +Carl Friedrich is a core developer since 2005, currently doing his PhD at the +Heinrich-Heine Universität Düsseldorf (Germany). He has worked on most aspects +of PyPy, from the core interpreter to the GC to the JIT. He has published +several papers about the inner workings of PyPy, presenting them at various +scientific conferences. Carl Friedrich is also interested in other dynamic +language implementation and was the original author of the Prolog +implementation. + +Carl Friedrich likes science fiction novels and sometimes plays the bassoon. + + +Antonio Cuni +============ + +.. image:: images/people/antocuni.png + +Antonio Cuni loves skiing, mountains and programming languages. He studied +Computer Science at the University of Genova (Italy), and then at the same +university he obtained his Ph.D. in Computer Science in 2010, with a +dissertation about the PyPy CLI JIT backend. He has been a core PyPy +developer since 2006, working in various areas including the "object oriented +backends" for the CLI and JVM, the RPython translation framework, the Python +interpreter and the JIT compiler generator. Apart from PyPy, he is the author of +other popular tools such as ``pdb++``. + +Benjamin Peterson +================= + +Both a PyPy and CPython core developer, Benjamin knows way too much about the +nooks and cranies of the Python language. He is driven by a fascination with +interpreters and compilers of all shapes and sizes. Around the PyPy project, he +tries to be generally useful and has taken on major projects including rewriting +PyPy's Python compiler and porting PyPy to Python 2.7. + +Alex Gaynor +=========== + +.. image:: images/people/alex.jpg + +Alex is software engineer living in Washington, DC. He's been a PyPy developer +since 2010, and has worked on many parts of the codebase, including the JIT +compiler's optimizers, the RPython translation toolchain, and the Python +interpreter. In addition to his work on PyPy, Alex is also the creator of +Topaz, a Ruby VM built on RPython and a core developer of Django (a Python web +framework) and CPython, as well as a retired member of the board of directors +of the Python Software Foundation. + +Håkan Ardö +========== + +.. image:: images/people/hakanardo.jpg + +Håkan Ardö received his master of science degree in electrical +engineering from Lund University in 2002. He specialized in +VLSI-design and Image Processing. He worked as a software +engineer at Axis Communications 2002-2003 before doing his +PhD at the Centre for Mathematical Sciences of Lund University +2003-2009 in the Mathematical Imaging Group. His thesis work consisted +of designing image processing algorithms for traffic surveillance, +aiming for a system that automatically measures the safety of an +intersection or road segment. He is currently working part-time as a +postdoc at the Centre for Mathematical Sciences of Lund University +continuing this work and part-time as CTO with a spinoff company +Cognimatics. His contributions to PyPy started 2010 and consists of +the array module as well as work on the JIT compiler's trace optimizers. + +Holger Krekel +================== + +.. image:: images/people/holger1.jpg + +Holger Krekel is a founder of the PyPy project and has participated in +PyPy core developement for several years as well as maintained much of +its infrastructure. He also is the author of the popular `py.test`_ and +`tox`_ testing tools as well as execnet_, a library for easily deploying +different interacting Python interpreters side by side. He helped +manage multiple PyPy funding contracts through his company merlinux and is a +PyPy representative within the Software Freedom Conservancy (SFC). He +holds a summa cum laude degree in computer science with a thesis about +artificial intelligence applied to the game of Go. As of 2011 he is on +another sabbatical-ish leave, caring for his newborn son, travelling +and pondering what comes next. Other than that he continues to care +for testing and some PyPy co-ordination bits behind the scene. + +.. _`py.test`: http://pytest.org +.. _`tox`: http://codespeak.net/tox +.. _`execnet`: http://codespeak.net/execnet + +Samuele Pedroni +=============== + +Samuele Pedroni got involved with PyPy almost at its inception in the +spring of 2003. One of the design contributors to PyPy, his help has +ranged from infrastructure and processes, through building out +RPython... optimizing the Python interpreter, to compressing resume +data in the last incarnation of the JIT compiler. Tempted away into the +application side of the software equation, these days he contributes +some words and wisdom to PyPy's paper writing. + + +Many more people +================ + +PyPy is and has always been an effort of many volunteers. Consult the `LICENSE`_ +file for details. + +.. _`LICENSE`: https://bitbucket.org/pypy/pypy/src/tip/LICENSE diff --git a/pages/performance.rst b/pages/performance.rst new file mode 100644 --- /dev/null +++ b/pages/performance.rst @@ -0,0 +1,376 @@ +.. title: Performance +.. slug: performance +.. date: 2019-12-28 16:14:02 UTC +.. tags: +.. category: +.. link: +.. description: + +.. contents:: + :depth: 1 + +This document collects strategies, tactics and tricks for making your +code run faster under PyPy. Many of these are also useful hints for +stock Python and other languages. For contrast, we also describe some +CPython (stock Python) optimizations that are not needed in PyPy. + + +================= + +.. _profiler: +.. _profiling: + +Profiling: vmprof +================= + +As a general rule, when considering performance issues, follow these +three points: first *measure* them (it is counter-productive to fight +imaginary performance issues); then *profile* your code (it is useless +to optimize the wrong parts). Only optimize then. + +PyPy 2.6 introduced vmprof_, a very-low-overhead statistical profiler. +The standard, non-statistical ``cProfile`` is also supported, and can be +enabled without turning off the JIT. We do recommend vmprof anyway +because turning on cProfile can distort the result (sometimes massively, +though hopefully this should not be too common). + +.. _vmprof: https://vmprof.readthedocs.org/ + + +===================== + +Optimization strategy +===================== + +These suggestions apply to all computer languages. They're here as +reminders of things to try before any Python or PyPy-specific tweaking. + +Build a regression-test suite +----------------------------- + +Before you start tuning, build a regression-test suite for your code. +This front-loads a significant amount of work, but it means you can +try lots of optimizations without worrying so much about introducing +functional bugs. + +Measure, don't guess +-------------------- + +Human beings are bad at guessing or intuiting where the hotspots in code are. +Measure, don't guess; use a profiler_ to pin down the 20% of the +code where the code is spending 80% of its time, then speed-tune that. + +Measuring will save you a lot of effort wasted on tuning parts of the code +that aren't actually bottlenecks. + +As you tune, re-profile frequently so you can see how the hottest spots +are shifting around. + +I/O-bound is different from compute-bound +----------------------------------------- + +Be aware of the difference between code that is compute-bound (slow +because it's doing a huge number of instructions) and code that is I/O +bound (slow because of disk or network delays). + +Expect to get most of your gains from optimizing compute-bound code. +It's usually (though not always) a sign that you're near the end of +worthwhile tuning when profiling_ shows that the bulk of the +application's time is spent on network and disk I/O. + +Tune your algorithms first +-------------------------- + +Generally, when your code is doing things that are O(n**2) or larger +in the size of your data set, the cost of those operations is going +to swamp any small gains you can pick up with the tricks we describe +here. + +Tune your algorithms first. It's time to think about applying our +list of micro-tuning tips *after* you think you've optimized out +intrinsically expensive operations. + +That said, be prepared for the possibility that you will discover +better-hidden algorithmic problems as you micro-tune. Likely +you will go through this cycle more than once. + +Focus on tight loops +-------------------- + +It's extremely common for high time costs to lurk within some +innocuous-looking code inside a tight loop - especially in code +that does something like a searching/matching/lookup operation +or any kind of graph traversal. + +Probably the most common kind of performance-killer in compute-bound +code is an O(n**2) operation that is disguised by being some sort of +O(n) lookup or match inside an O(n) loop. + +Another common time-sink is relatively expensive common-setup +operations that are performed inside tight loops but could be moved +to before they start. (For a representative case of this, see the +micro-tuning tip on regexp compilation.) + +Smaller is faster +----------------- + +Modern computers have multiple levels of memory caching, some directly +on the processor chip. Causing a cache miss at any level incurs a +performance penalty proportional to random-access time for the next +outward (and much slower) layer of cache. + +Accordingly, smaller is faster. Programs or routines with a small +enough working set to fit inside a fast cache will be as fast as +that cache is. To make your code fast, reduce the length of the +series of Python or JIT-compiler opcodes it generates by making +it simpler. + +The tradeoff here is that algorithmic tuning often trades time for +space - that is, it increases the size of an algorithm's working set +by including pre-computations or tables or reverse maps in order to +avoid O(n**2) operations. + +It's impossible to predict in advance where the sweet spot in that +tradeoff will be. You have to try different things and measure - +which takes us right back to "Measure, don't guess". And another +function of your regression test suite can be as a speed benchmark. + + +================= + +Micro-tuning tips +================= + +These are in no particular order. + +Keep it simple +-------------- + +Simple is better than complex. The PyPy JIT is not very smart; the +simpler your code is the better it will run. Here again, though, you face +a tradeoff: you may need to pay with more algorithmic complexity in order +to avoid brute-force operations that are O(n**2) or worse. + +Write plain-vanilla code in plain-vanilla ways. The PyPy JIT has many +productions that optimize a common usage pattern against an uncommon +usage pattern. + +Global variables +---------------- + +In CPython, global variables and functions (including package imports) +are much more expensive to reference than locals; avoid them. (This +is also good modularity practice). + +The cost of CPython global references is high enough that, for example, if you +have code in a frequently-visited inner loop that uses int() a lot, it +may be worthwhile to create a local copy of the reference with "int = +int" in an enclosing block. + +However, this in *not* true in JITted PyPy code. The "int = int" hack +won't buy you performance, it's just an extra copy. The modularity +reason for avoiding globals are still valid. + +Regular expressions +------------------- + +Regular-expression compilation is expensive. If the regexp pattern in +a search, match, or replace operation is static (doesn't mutate at +runtime) refactor so it's only done once. + +If the regexp compilation is in a class method, consider doing it as +the initializer of a regexp-valued static (shared) class member and +using that class member in your operation. + +If the regexp compilation is in a free function, consider moving it +to module level and referencing the resulting regexp object +(but see the warning above about global variables). + +Old- vs. new-style classes +-------------------------- + +New-style classes allow faster attribute access and take up less core +per instance than old-style classes. Much of this advantage may be +lost, however, if attribute names are not constant. For example: x.a += y or even setattr(x, 'a', y) will be much faster than a dynamic +version: setattr(x, 'a' + some_variable, y). + +Classes that inherit from both new- and old-style classes are +*extremely* slow; avoid at all costs. + +In PyPy, isinstance() called against an old-style class was very slow +until 2.0. + +String concatenation is expensive +---------------------------------- + +In CPython, you may want to replace: + +.. code-block:: python + + s = head + body + maybe + tail + +with the admittedly less readable: + +.. code-block:: python + + s = "%(head)s%(body)s%(maybe)s%(tail)s" % locals() + +or even: + +.. code-block:: python + + s = "{head}{body}{maybe}{tail}".format(**locals()) + +Both of the latter forms avoid multiple-allocation overhead. +But PyPy's JIT makes the overhead of intermediate concatenations +go away in linear code that keeps the number of concatenations +small, bound and constant. (And ``locals()`` is rather slow +with PyPy's JIT.) + +On the other hand, in code like this with a string-valued foo() function: + +.. code-block:: python + + for x in mylist: + s += foo(x) + +the JIT cannot optimize out intermediate copies. This code is +actually quadratic in the total size of the mylist strings due to +repeated string copies of ever-larger prefix segments. (Such code +is always fine for bytearrays, because in this case ``+=`` is an +in-place operation.) + +This: + +.. code-block:: python + + parts = [] + for x in mylist: + parts.append(foo(x)) + s = "".join(parts) + +can be much faster because all the string concatenation in the last +line creates exactly one new string object with one C-level copy +sequence (and list operations are relatively cheap). + +Frame introspection and tracing are slow +---------------------------------------- + +Certain function calls can disable PyPy's speed options over +stretches of surrounding code called "JIT scopes". + +A JIT like PyPy's works based on the assumption that the only thing +worth optimizing are loops that are executed often. Whenever the +interpreter enters a loop in the interpreted program, the JIT records +what the interpreter does, creating a trace. This trace is optimized, +compiled to machine code and executed when the loop is hit with the +conditions observed during tracing. This trace is one kind of JIT scope. + +Another kind of JIT scope that matters is a function, considered as +a unit for inlining. + +Note that a JIT scope is a run-time phenomenon, not a compile-time +one. It's not confined by source-code module boundaries. A library- +or foreign-module call in a frequently-called loop or inlined function +will be part of its JIT scope. + +locals(), globals(), sys._getframe(), sys.exc_info(), and sys.settrace +work in PyPy, but they incur a performance penalty that can be huge by +disabling the JIT over the enclosing JIT scope. + +*(Thanks Eric S. Raymond for the text above)* + + +======================= +Insider's point of view +======================= + +This section describes performance issues from the point of view of +insiders of the project; it should be particularly interesting if you +plan to contribute in that area. + +One of the goals of the PyPy project is to provide a fast and compliant +python interpreter. Some of the ways we achieve this are by providing a +high-performance garbage collector (GC) and a high-performance +Just-in-Time compiler (JIT). Results of comparing PyPy and CPython can +be found on the `speed website`_. Those benchmarks are not a random +collection: they are a combination of real-world Python programs --- +benchmarks originally included with the (now dead) Unladen Swallow +project --- and benchmarks for which we found PyPy to be slow (and improved). +Consult the descriptions of each for details. + +The JIT, however, is not a magic bullet. There are several characteristics +that might surprise people who are not used to JITs in +general or to the PyPy JIT in particular. The JIT is generally good at +speeding up straight-forward Python code that spends a lot of time in the +bytecode dispatch loop, i.e., running actual Python code --- as opposed +to running things that only are invoked by Python code. Good +examples include numeric calculations or any kind of heavily +object-oriented program. Bad examples include doing computations with +large longs --- which is performed by unoptimizable support code. When the +JIT cannot help, PyPy is generally slower than CPython. + +More specifically, the JIT is known not to work on: + +* **Tests**: The ideal unit tests execute each piece of tested code + once. This leaves no time for the JIT to warm up. + +* **Really short-running scripts**: A rule of thumb is if something runs below + 0.2s the JIT has no chance, but it depends a lot on the program in question. + In general, make sure you warm up your program before running benchmarks, if + you're measuring something long-running like a server. The time required + to warm up the JIT varies; give it at least a couple of seconds. (PyPy's + JIT takes an especially long time to warm up.) + +* **Long-running runtime functions**: These are the functions provided + by the runtime of PyPy that do a significant amount of work. + PyPy's runtime is generally not as optimized as CPython's and we expect those + functions to take somewhere between the same time as CPython to twice as long. + This includes, for example, computing with longs, or sorting large lists. + A counterexample is regular expressions: although they take time, they + come with their own JIT. + +Unrelated things that we know PyPy to be slow at (note that we're probably +working on it): + +* **CPython C extension modules**: Any C extension module recompiled + with PyPy takes a very large hit in performance. PyPy supports C + extension modules solely to provide basic functionality. + If the extension module is for speedup purposes only, then it + makes no sense to use it with PyPy at the moment. Instead, remove it + and use a native Python implementation, which also allows opportunities + for JIT optimization. If the extension module is + both performance-critical and an interface to some C library, then it + might be worthwhile to consider rewriting it as a pure Python version + that uses CFFI_ for the interface. + +* **Missing RPython modules**: A few modules of the standard library + (like ``csv`` and ``cPickle``) are written in C in CPython, but written + natively in pure Python in PyPy. Sometimes the JIT is able to do a + good job on them, and sometimes not. In most cases (like ``csv`` and + ``cPickle``), we're slower than CPython, with the notable exception of + ``json`` and ``heapq``. + +* **Abuse of itertools**: The itertools module is often "abused" in the + sense that it is used for the wrong purposes. From our point of view, + itertools is great if you have iterations over millions of items, but + not for most other cases. It gives you 3 lines in functional style + that replace 10 lines of Python loops (longer but arguably much easier + to read). The pure Python version is generally not slower even on + CPython, and on PyPy it allows the JIT to work much better --- simple + Python code is fast. The same argument also applies to ``filter()``, + ``reduce()``, and to some extend ``map()`` (although the simple case + is JITted), and to all usages of the ``operator`` module we can think + of. + +* **Ctypes**: Ctypes is slower than on CPython. Consider CFFI_ instead, + which has special paths inside the JIT. + +We generally consider things that are slower on PyPy than CPython to be bugs +of PyPy. If you find some issue that is not documented here, +please report it to our `bug tracker`_ for investigation. + +.. _`bug tracker`: https://bitbucket.org/pypy/pypy/issues?status=new&status=open +.. _`speed website`: http://speed.pypy.org +.. _CFFI: http://cffi.readthedocs.org/ _______________________________________________ pypy-commit mailing list [email protected] https://mail.python.org/mailman/listinfo/pypy-commit
