On Fri, Jan 28, 2022 at 6:28 PM Guido van Rossum <gu...@python.org> wrote:
> I think we will get *one* chance in the next decade to get it right. Whether
> that's HPy or evolution of the C API I'm not sure.
Would you mind to elaborate? Which risk do you expect from switching
to HPy and from fixing the C API (introducing incompatible C API
changes)?
For me, promoting HPy and evolution of the C API are complementary,
can and must done in parallel for me. As I explained in PEP 674, while
HPy does help C extensions writers, it doesn't solve any problem for
CPython right now. CPython is still blocked by implementation details
leaked throught the C API that we must still maintain for a few more
years.
> Victor, am I right that the (some) stable ABI will remain important because
> projects don't have resources to build wheels for every Python release? If a
> project does R releases per year for P platforms that need to support V
> versions of Python, they would normally have to build R * P * V wheels. With
> a stable ABI, they could reduce that to R * P. That's the key point, right?
There are different use cases.
1) First, my main worry is that we put a high pressure on maintainers
of most important Python dependencies before the next of a new Python
version, because we want them to handle the flow of incompatible C API
changes before the final Python 3.x versions is released, to get them
available when Python 3.x final is released.
It annoys core developers who cannot change things in Python without
getting an increasing number of complains about a large number of
broken packages, sometimes with a request to revert.
It annoys C extensions maintainers who have to care about Python alpha
and beta releases which are not convenient to use (ex: not available
in Linux distributions). Moreover, it became common to ask multiple
changes and multiple releases before a Python final release, since
more incompatible changes are introduced in Python (before the beta1).
2) Second, as you said, the stable ABI reduces the number of binary
packages which have to be built. Small projects with a little team
(ex: a single person) don't have resources to set up a CI and maintain
it to build all these packages. It's doable, but it isn't free.
--
The irony of the situation is that we must break the C API (hiding
structures is technically an incompatible change)... to make the C API
stable. Breaking it now to make it stable later.
We already broke the C API many times in the past. The difference here
is that changes are done in the purpose of bending it towards the
limited C API and the stable ABI.
My expectation is that replacing frame->f_code with PyFrame_GetCode()
only has to be done exactly once: this API is not going this change.
Sadly, the changes are not limited to frame->f_code, more changes are
needed. For example, for PyFrameObject, accesses to every structure
member must have to go through a function call (getter or setter
function). Hopefully, only a small number of member are used by C
extensions.
The tricky part is to think about the high level API ("use cases")
rather than just adding functions doing "return struct->member" and
"struct->member = new_value". The PyThreadState_EnterTracing() and
PyThreadState_LeaveTracing() functions added to Python 3.11 are a good
example: the API is "generic" and the implementation changes 2
structure members, not a single one.
In practice, what I did since Python 3.8 is to introduce a small
number of C API changes per Python versions. We tried the "fix all the
things at once" approach (!!!) with Python 3, and it... didn't go
well. All C extensions had to suddenly write their own compatibility
layer for a large number of C API functions (ex: replace PyInt_xxx
with PyLong_xxx, without losing Python 2 support!). The changes that
I'm introducing in the C API usually impact less than 100 extensions
in total (usually, I would say between 10 and 25 per Python version,
but it's hard to measure exactly).
> Can HPy do that?
I wish more projects are incrementally rewritten with Cython, cffi,
pybind11 and HPy, and so slowly move away using directly the C API.
Yes, HPy support an "universal build" mode which allows to only build
a C extension once, and use it on multiple *CPython* versions *and*
(that's the big news!) multiple *PyPy* versions! I even heard that it
also brings GraalPython support for free ;-)
Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/4I7N3SCBWIJCYXE3WGCQBMRTATVGIHY6/
Code of Conduct: http://python.org/psf/codeofconduct/
