Hi all, See below for a proposal for a major change to the public NumPy C headers that I'd like to work on. There are some questions I have at the end that I'd appreciate comments on from interested parties.
Thanks for your help, Nathan Goldbaum ## Motivation Python 3.15 will make it possible to build extensions using the limited API on the free-threaded build. See PEP 803 (https://peps.python.org/pep-0803/) and accompanying discourse thread for more details about this effort in CPython. The free-threaded build and GIL-enabled build have different layouts for PyObject, so this new ABI will abstract over this difference by making PyObject opaque. ## The proposal NumPy exposes several C structs that extend PyObject (e.g. they are statically defined using PyObject_HEAD), so all of these objects will need updates. I'd like to update NumPy's C headers to deprecate directly accessing structs that extend PyObject starting in the NumPy 2.5 C API. In particular, I'd like to make the following structs opaque in the NumPy C API starting in NumPy 2.5: * PyArrayObject * PyArray_Descr * PyArray_Chunk * PyArrayIterObject * PyArrayMultiIterObject * PyArrayNeighborhoodIterObject * All of the array scalar types (e.g. PyBoolScalarObject). I'll also need to add new accessor macros to access the data in these fields. I may also need setter macros if we need support mutating any of these fields. The descriptor object in particular needs support for mutability to enable setup of user DTypes. I went ahead and implemented this change for the descriptor struct, see https://github.com/numpy/numpy/pull/30744. So far Sebastian and Matti have indicated they like the approach. It's a big change (almost 1,000 LOC diff) but the vast majority of changes are mechanical switches to the new accessor macros. I expect making this change for the other objects will not be as impactful codebase-wide. If we do make this an API change and deprecate the old API, I think the biggest impact will be on packages implementing user DTypes, since they statically extend the PyArray_Descr struct. I expect all other users to be able to trivially use accessor macros instead. ## Next steps I'd like to make sure this plan is OK with the mailing list audience before I do any more work towards this. I'm not sure if I'd like to hard-deprecate access to these fields, because I'll need to add new accessor macros to NumPy's headers that people will need to copy/paste to be able to set NPY_NO_DEPRECATED_API to NPY_2_5_API_VERSION and still build using older NumPy releases. Unfortunately due to the way NumPy's C deprecations work, this may cause lots of noisy messages in people's build logs about use of deprecated APIs that they can't easily fix in practice. The alternative is to only make these objects opaque during opaque PyObject builds. I think I'd also like to deprecate access in internal builds as I've done in my PR, since that forces us to actually use the opaque object API and sets us on the path towards publicly deprecating the old API.
_______________________________________________ NumPy-Discussion mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3//lists/numpy-discussion.python.org Member address: [email protected]
