On Tue, 2020-04-21 at 16:21 +0200, Victor Stinner wrote: > Le mar. 21 avr. 2020 à 00:50, Nathaniel Smith <n...@pobox.com> a écrit > : <snip> > > > > tl;dr: accepting PEP 554 is effectively a C API break, and will > > force > > many thousands of people worldwide to spend many hours wrangling > > with > > subinterpreter support. > > I fail to follow your logic. When the asyncio PEP was approved, I > don't recall that suddenly the whole Python community started to > rewrite all projects to use coroutines everywhere. I tried hard to > replace eventlet with asyncio in OpenStack and I failed because such > migration was a very large project with dubious benefits (people > impacted by eventlet issues were the minority).
Sure, but this is very different. You can still use NumPy in a project using asyncio. You are _not_ able to use NumPy in a project using subinterpreters. Right now, I have to say as soon as the first bug report asking for this is opened and tells me: But see PEP 554 you should support it! I would be tempted to put on the NumPy Roadmap/Vision that no current core dev will put serious efforts into subinterpreters. Someone is bound to be mad. Basically, if someone wants it in NumPy, I personally may expect them to be prepared to invest a year worth of good dev time [1]. Maybe that is pessimistic, but your guess is as good as mine. At normal dev-pace it will be at least a few years of incremental changes before NumPy might be ready (how long did it take Python?)? The PEP links to NumPy bugs, I am not sure that we ever fixed a single one. Even if, the remaining ones are much larger and deeper. As of now, the NumPy public API has to be changed to even start supporting subinterpreters as far as I aware [2]. This is because right now we sometimes need to grab the GIL (raise errors) in functions that are not passed GIL state. This all is not to say that this PEP itself doesn't seem harmless. But the _expectation_ that subinterpreters should be first class citizens will be a real and severe transition burden. And if it does not, the current text of the PEP gives me, as someone naive about subinterpreters, very few reasons why I should put in that effort or reasons to make me believe that it actually is not as bad a transition as it seems. Right now, I would simply refuse to spend time on it. But as Nathaniel said, it may be worse if I did not refuse and in the end only a handful of users get anything out of my work: The time is much better spend elsewhere. And you, i.e. CPython will spend your "please fix your C- extension" chips on subinterpreters. Maybe that is the only thing on the agenda, but if it is not, it could push other things away. Reading the PEP, it is fuzzy on the promises (the most concrete I remember is that it may be good for security relevant reasons), which is fine, because the goal is "experimentation" more than use? So if its more about "experimentation", then I have to ask, whether: 1. The PEP can state that more obviously, it wants to be provisionally/experimentally accept? So maybe it should even say that that extension modules are not (really) encouraged to transition unless they feel a significant portion of their users will gain. 2. The point about developing it outside of the Python standard lib should be considered more seriously. I do not know if that can be done, but C-API additions/changes/tweaks seem a bit orthogonal to the python exposure? So maybe it actually is possible? As far as I can tell, nobody can or _should_ expect subinterpreters to actually run most general python code for many years. Yes, its a chicken-and-egg problem, unless users start to use subinterpreters successfully, C-extensions should probably not even worry to transition. This PEP wants to break the chicken-and-egg problem to have a start, but as of now, as far as I can tell, it *must not* promise that it will ever work out. So, I cannot judge the sentiment or subinterpreters. But it may be good to make it *painfully* clear what you expect from a project like NumPy in the next few years. Alternatively, make it painfully clear that you possibly even discourage us from spending time on it now, if its not straight forward. Those using this module are on their own for many years, probably even after success is proven. Best, Sebastian [1] As of now, the way I see it is that I could not even make NumPy (and probably many C extensions) work, because I doubt that the limited API has been exercised enough [2] and I am pretty sure it has holes. Also the PEP about passing module state around to store globals efficiently seems necessary, and is not in yet? (Again, trust: I have to trust you that e.g. what you do to make argument parsing not have overhead in argument clinic will be something that I can use for similar purposes within NumPy) [2] I hope that we will do (many) these changes for other reasons within a year or so, but they go deep into code barely touched in a decade. Realistically, even after the straight forward changes (such as using the new PEPs for module initialization), these may take up an additional few months of dev time (sure, get someone very good or does nothing else, they can do it much quicker maybe). So yes, from the perspective of a complex C-extension, this is probably very comparable to the 2to3 change (it happened largely before my time though). [3] E.g. I think I want an ExtensionMetaClass, a bit similar as an ABC, but I would prefer to store the data in a true C-slot fashion. The limited API cannot do MetaClasses correctly as far as I could tell and IIRC is likely even a bit buggy. Are ExtensionMetaClasses crazy? Maybe, but PySide does it too (and as far as I can tell, they basically get away with it by a bit of hacking and relying on Python implementation details. > > When asyncio landed in Python 3.4, a few people started to experiment > it. Some had a bad experience. Some others were excited and put a few > applications in production. > > Even today, asyncio didn't replace threads, multiprocessing, > concurrent.futures, etc. There are even competitor projects like > Twisted, trio and curio! (Also eventlet and gevent based on greenlet > which is a different approach). I only started to see very recently > project like httpx which supports both blocking and asynchronous API. > > I see a slow adoption of asyncio because asyncio solves very specific > use cases. And that's fine! > > I don't expect that everyone will suddenly spend months of work to > rewrite their C code and Python code to be more efficient or fix > issues with subinterpreters, until a critical mass of users proved > that subinterpreters are amazing and way more efficient! > > Victor > -- > Night gathers, and now my watch begins. It shall not end until my > death. > _______________________________________________ > Python-Dev mailing list -- python-dev@python.org > To unsubscribe send an email to python-dev-le...@python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/3MK2NANMOJFHKJWELGI2ZD74K2ZYJAOD/ > Code of Conduct: http://python.org/psf/codeofconduct/ _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/ONEVR72KRC6VZR3X47BF5XOD5ZAVDQTZ/ Code of Conduct: http://python.org/psf/codeofconduct/
