Hi, What do you think of promoting the pythoncapi_compat project that I'm introducing below in the "C API: Porting to Python 3.10" section of What's New In Python 3.10?
Should this project be moved under the GitHub psf organization to have a more "future proof" URL? I would like to promote this project to prepare C extensions maintainers for the following incompatible C API change (Py_TYPE) that I would like to push into Python 3.11: https://github.com/python/cpython/pull/26493 (Py_REFCNT was already converted to a static inline function in Python 3.10.) I already made this Py_TYPE change in Python 3.10, but I had to revert it since it broke too many projects. Since last year, I upgraded most of these broken projects, I created the pythoncapi_compat project, and I succeeded to use upgrade_pythoncapi.py script and copy the pythoncapi_compat.h header file in multiple C extensions. C extensions written with Cython are not affected. I already fixed Cython last year to emit code compatible with my incoming incompatible change. If it's not done yet, you only have to regenerate the C files using a recent Cython version. -- I wrote a new script which adds Python 3.10 support to your C extensions without losing Python 2.7 support: https://github.com/pythoncapi/pythoncapi_compat To add Python 3.10 support to your C extension, go to its source directory and run: /path/to/upgrade_pythoncapi.py . It upgrades all C files (.c) in the current directory and subdirectories. For example, it replaces "op->ob_type" with "Py_TYPE(op)". It creates an ".old" copy of patched files. Use the -o option to select operations: * -o Py_TYPE: only replace "obj->ob_type" with "Py_TYPE(obj)". * -o all,-PyMem_MALLOC: run all operations, but don't replace PyMem_MALLOC(...) with PyMem_Malloc(...). -- The upgrade_pythoncapi.py script relies on the pythoncapi_compat.h header file that I wrote to provide recent Python 3.9-3.11 C functions on old Python versions. Examples: Py_NewRef() and PyThreadState_GetFrame(). Functions are implemented as simple static inline functions to avoid requiring to link your extension to a dynamic library. You can already use the new Py_NewRef() and Py_IsNone() Python 3.10 functions in your projects without losing support for Python 2.7-3.9! -- The script also replaces "frame->f_back" with "_PyFrame_GetBackBorrow(frame)". The _PyFrame_GetBackBorrow() function doesn't exist in the Python C API, it's only provided by pythoncapi_compat.h to ease the migration of C extensions. I advise you to replace _PyFrame_GetBackBorrow() (borrowed reference) with PyFrame_GetBack() (strong reference). -- This project is related to my PEP 620 "Hide implementation details from the C API" which tries to make the C API more abstract to later allow to implement new optimization in CPython and to make other Python implementations like PyPy faster when running C extensions. Article on the creation of the pythoncapi project: https://vstinner.github.io/pythoncapi_compat.html The main drawback of this project is that it uses regular expressions to parse C code. Such "parser" can miss C code which has to be patched manually. In my experience, additional manual changes are really rare and take less than 1 minute on a very large C extension like numpy. -- This project only targets extension modules written in C by using directly the "Python.h" API. I advise you to use Cython or HPy to no longer be bothered with incompatible C API changes at every Python release ;-) * https://cython.org/ * https://hpy.readthedocs.io/ I hope that my script will facilitate migration of C extensions to HPy. Victor -- Night gathers, and now my watch begins. It shall not end until my death. _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/KHDZGCNOYEDUTSPAATUDP55ZSSQM5RRC/ Code of Conduct: http://python.org/psf/codeofconduct/
