Author: Antonio Cuni <anto.c...@gmail.com> Branch: extradoc Changeset: r5966:48e199e316c7 Date: 2019-12-17 16:41 +0100 http://bitbucket.org/pypy/extradoc/changeset/48e199e316c7/
Log: start a blog post about the hpy sprint diff --git a/blog/draft/2019-12-hpy-sprint.rst b/blog/draft/2019-12-hpy-sprint.rst new file mode 100644 --- /dev/null +++ b/blog/draft/2019-12-hpy-sprint.rst @@ -0,0 +1,165 @@ +HPy kick-off sprint report +=========================== + +Recently Antonio, Armin and Ronan had a small internal sprint in the beautiful +city of Gdańsk to kick-off the development of HPy. Here is a brief report of +what it has been accomplished during the sprint. + +What is HPy? +------------ + +The TL;DR answer is "a better way to write C extensions for Python". + +The idea of HPy was born during EuroPython 2019 in Basel, where there was an +informal meeting which included core developers of PyPy, CPython (Victor +Stinner and Mark Shannon) and Cython (Stefan Behnel). + +All of us agreeded that the current design of the CPython C API is problematic +for various reasons and, in particular, because it is too tied to the current +internal design of CPython. The end result is that: + + - alternative implementations of Python (such as PyPy, but not only) have a + `hard time` to load and execute existing C extensions; + + - CPython itself is unable to change some of its internal implementation + details without breaking the world. For example, as of today it would be + impossible to switch from using reference counting to using a real GC, + which in turns make it hard for example to remove the GIL, as gilectomy_ + attempted. + +HPy tries to address these issues by following two major design guidelines: + + 1. objects are referenced and passed around using opaque handles, which are + similar to e.g. file descriptors in spirit. Multiple, different handles + can point to the same underlying object, handles can be duplicated and + each handle must be closed independently. + + 2. The internal data structures and C-level layout of objects are not + visible nor accessible using the API, so each implementation if free to + use what fits best. + +The other major design goal of HPy is to allow an incremental +transition/porting, so existing module can migrate their codebase one method +at a time. Moreover, Cython eventually will generate HPy code, so extension +module written in Cython will be able to benefit from HPy automatically. + +More details can be found in the README of the official `HPy repository`_. + +.. _`hard time`: https://morepypy.blogspot.com/2018/09/inside-cpyext-why-emulating-cpython-c.html +.. _gilectomy: https://pythoncapi.readthedocs.io/gilectomy.html +.. _`HPy repository`: https://github.com/pyhandle/hpy + + +CPython and universal target ABI +--------------------------------- + +When compiling an HPy extension you can choose two different target ABI: + + - **CPython ABI**: in this case, ``hpy.h`` contains a set of macros and + static inline functions which translates at compilation time the HPy API + into the standard C-API: the compiled module will have no performance + penalty and it will have an filename like + ``foo.cpython-37m-x86_64-linux-gnu.so``. + + - **Universal HPy ABI**: as the name implies, extension modules compiled + this way are "universal" and can be loaded unmodified by multiple Python + interpreters and version. Moreover, it will be possible to dynamically + enable a special debug mode which will make it easy to find e.g. unclosed + handles or memory leaks, **without having to recompile the extension**. + + +Universal modules can be loaded **also** on CPython, thanks to the +``hpy_universal`` module which is under development: because of an extra layer +of indirection, extensions compiled with the universal ABI will face a small +performance penalty compared to the ones using the CPython ABI. + +This setup gives several benefits: + + - extension developers can use the extra debug features given by the + Universal ABI with no need to use a special debug version of Python + + - projects which need the maximum level of performance can compile their + extension for each relevant version of CPython, as they are doing now + + - projects for which runtime speed is less important will have the choice of + distributing a single binary which will work on any version of Python + + + +A simple example +----------------- + +The HPy repo contains a `proof of concept`_ module. Here is a simplified +version which illustrates how an HPy module looks like: + +.. sourcecode:: C + + #include "hpy.h" + + HPy_DEF_METH_VARARGS(add_ints) + static HPy add_ints_impl(HPyContext ctx, HPy self, HPy *args, HPy_ssize_t nargs) + { + long a, b; + if (!HPyArg_Parse(ctx, args, nargs, "ll", &a, &b)) + return HPy_NULL; + return HPyLong_FromLong(ctx, a+b); + } + + + static HPyMethodDef PofMethods[] = { + {"add_ints", add_ints, HPy_METH_VARARGS, ""}, + {NULL, NULL, 0, NULL} + }; + + static HPyModuleDef moduledef = { + HPyModuleDef_HEAD_INIT, + .m_name = "pof", + .m_doc = "HPy Proof of Concept", + .m_size = -1, + .m_methods = PofMethods + }; + + + HPy_MODINIT(pof) + static HPy init_pof_impl(HPyContext ctx) + { + HPy m; + m = HPyModule_Create(ctx, &moduledef); + if (HPy_IsNull(m)) + return HPy_NULL; + return m; + } + + +People who are familiar with the current C-API will surely notice lots of +similarities. The biggest differences are: + + - Instead of ``PyObject *``, objects have the type ``HPy``, which as + explained above represents a handle. + + - You need to explicitly pass an ``HPyContext`` around: the intent is + primary to be future-proof and make it easier to implement things like + sub- interpreters. + + - ``HPy_METH_VARARGS`` is implemented differently than CPython's + ``METH_VARARGS``: in particular, these methods receive an array of ``HPy`` + and its length, instead of a fully constructed tuple: passing a tuple + makes sense on CPython where you have it anyway, but it might be an + unnecessary burden for alternate implementations. Note that this is + similar to the new `METH_FASTCALL` which was introduced in CPython. + + - HPy relies a lot on C macros, which most of the time are needed to support + the CPython ABI compilation mode. For example, ``HPy_DEF_METH_VARARGS`` + expands into a trampoline which has the correct C signature that CPython + expects (i.e., ``PyObject (*)(PyObject *self, *PyObject *args)``) and + which calls ``add_ints_impl``. + + +.. _`proof of concept`: https://github.com/pyhandle/hpy/blob/master/proof-of-concept/pof.c +.. _`METH_FASTCALL`: https://www.python.org/dev/peps/pep-0580/ + + +Sprint report and current status +--------------------------------- + +XXX finish me _______________________________________________ pypy-commit mailing list pypy-commit@python.org https://mail.python.org/mailman/listinfo/pypy-commit
