On Tue, Apr 29, 2025 at 11:24 AM PIERRE AUGIER < pierre.aug...@univ-grenoble-alpes.fr> wrote:
> Dear Numpy community members and Numpy developers, > > This email is to get the points of view of the Numpy community members and > developers about a subject that I find very important. I'm going to > introduce it just in few lines so I write few statements without backing > them with proper arguments and without giving links and word definitions. I > assume that most people registered to the numpy-discussion list are at > least familiar to the subject. > Thanks for thinking about this Pierre. > I think getting proper compatibility of the Python scientific/data stack > with fast Python interpreters is very important for the long-term future of > Python for science and data. > I'm not sure it is, it wouldn't rank high on my wish list. PyPy is nearing its end-of-life, and GraalPy is really only for Java devs it seems - I've never seen it in the real world. With CPython performance being worked hard on, the performance gap is shrinking every year. More importantly, none of these efforts (including the "faster CPython" project), seem critical to numerical/scientific users. We're still talking about pure Python code that gets maybe up to 5x faster, while the gains from doing things in compiled languages are a lot higher. So the benefits are more important for small packages if it moves the threshold at which it becomes necessary for them to write zero extension modules. For core libraries like NumPy, pure Python performance isn't super critical. When thinking about overall performance improvements, I'd say that the two most promising large developments are: (1) making it easier to use accelerator libraries (PyTorch, CuPy et al.), and (2) free-threaded CPython for enabling Python-level threading. > Nowadays, fast Python implementations like PyPy and GraalPy are > > - complicated to use (need local builds of wheels) > - slow as soon as the code involves Python extensions (because extensions > work through an emulation layer, like cpyext in PyPy) > > Therefore, these fast Python implementations are not as popular as they > should be. However, they are really great tools which should be used in > particular for scientific/data applications. > > Unfortunately, if we just follow the pace of the CPython C API evolution, > we will basically never have an ecosystem natively compatible with fast > Python implementations. > > The more I think and learn about this subject, the more I think that Numpy > has to stop using directly the CPython C API and to be ported to HPy > (though there are other alternatives - in particular Nanobind and Cython - > that could be discussed). Numpy 1 has been ported to HPy but unfortunately, > there have been deep changes with Numpy 2 (and Meson) so it seems that one > should restart the porting. > > Moreover, unfortunately, HPy does not currently receive as much care as it > should. > > It seems to me that the project of fixing the roots of the Python > ecosystem has to be relaunched. I think that the dynamics has to come from > the Python scientific/data community and in particular Numpy. It is > unfortunately outside of the C API working group's scope (see > https://discuss.python.org/t/c-api-working-group-and-plan-to-get-a-python-c-api-compatible-with-alternative-python-implementations/89477 > ). > > It seems to me that it is necessary (and possible) to get some founding > for such an impactful project so that we can get people working on it. > > I'd like to write a long and serious text on this subject, and I first try > to get the points of view of the different people and projects involved. > > I guess I should write explicit questions: > > - What do you think about the project of fixing the Python scientific/data > stack so that it becomes natively compatible (hence fast and convenient, > with easy installations) with alternative and fast Python interpreters? > - Do you have points of view on how this should be done, technically > (HPy?, something else?) and on other aspects (community, NEP?, founding, > ...). > - Anything else interesting on this subject? > Having HPy or something like it will be very nice and lead ot long-term benefits. The problem with HPy seems to be more a social one at this point: if CPython core devs don't want to adopt it but do their own "make the C API more opaque" strategy, then more effort on HPy isn't going to help. If you're going to dig into this more, I suggest trying to get a very good sense of what the CPython core dev team, and in particular its C API Workgroup, is thinking/planning. That will inform whether the right strategy is to help their efforts along, or work on HPy. Cheers, Ralf > > Best regards, > Pierre > > -- > Pierre Augier - CR CNRS http://www.legi.grenoble-inp.fr > LEGI (UMR 5519) Laboratoire des Ecoulements Geophysiques et Industriels > BP53, 38041 Grenoble Cedex, France tel:+33.4.56.52.86.16 > _______________________________________________ > NumPy-Discussion mailing list -- numpy-discussion@python.org > To unsubscribe send an email to numpy-discussion-le...@python.org > https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ > Member address: ralf.gomm...@googlemail.com >
_______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
