[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
Thanks for the thoughtful post! I'm going to address some of your comments here and some in a separate discussion in the next few days. On Wed, Apr 29, 2020 at 10:36 AM Sebastian Berg wrote: > While I still think it is probably not part of PEP 554 as such, I guess > it needs a full blown PEP on its own. Saying that Python should > implement subinterpreters. (I am saying "implement" because I believe > you must consider subinterpreters basically a non-feature at this time. > It has neither users nor reasonable ecosystem support.) FWIW, at this point it would be hard to justify removing the existing public subinterpreters C-API. There are several large public projects using it and likely many more private ones we do not know about. That's not to say that alone justifies exposing the C-API, of course. :) > In many ways I assume that a lot of the ground work for subinterpreters > was useful on its own. There has definitely been a lot of code health effort related to the CPython runtime code, partly motivated by this project. :) > But please do not underestimate how much effort > it will take to make subinterpreters first class citizen in the > language! If you are talking about on the CPython side, most of the work is already done. The implementation of PEP 554 is nearly complete and subinterpreter support in the runtime has a few rough edges to buff out. The big question is the effort it will demand of the Python community, which is the point Nathaniel has been emphasizing (understandably). > Believe me, I have been there and its tough to write these documents > and then get feedback which you are not immediately sure what to make > of. > Thus, I hope those supporting the idea of subinterpreters will help you > out and formulate a better framework and clarify PEP 554 when it comes > to the fuzzy long term user-impact side of the PEP. FYI, I started working on this project in 2015 and proposed PEP 554 in 2017. This is actually the 6th round of discussion since then. :) -eric ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/SIKH5NK6B67BLLVHDRAMK64PMO6EZ5VI/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
Thanks for the great insights into PyObjC! On Wed, Apr 29, 2020 at 9:02 AM Ronald Oussoren wrote: > I don’t know how much the move of global state to per-interpreter state > affects extensions, other than references to singletons and static types. That's the million dollar question. :) FYI, one additional challenge is when an extension module depends on a third-party C library which itself keeps global state which might leak between subinterpreters. The Cryptography project ran into this several years ago with OpenSSL and they were understandably grumpy about it. > But with some macro trickery that could be made source compatible for > extensions. Yeah, that's one approach that we've discussed in the past (e.g. at the last core sprint). -eric ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/H3RCS47ZUITKKXR3BVYOPXNXBZYF5ZN4/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
On Wed, Apr 29, 2020 at 6:27 AM Julien Salort wrote: > If your proposal leads to an intelligible actual error, and a clear > warning in the documentation, instead of a silent crash, this sounds > like progress, even for those packages which won't work on > subinterpreters anytime soon... That's helpful. Thanks! -eric ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/BURRXBY24URPSZXRIB7OHYCEBY2G4U67/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
On Wed, Apr 29, 2020 at 1:52 AM Paul Moore wrote: > One thing that isn't at all clear to me here is that when you say > "Subinterpreters run all Python code", do you *just* mean the core > language? Or the core language plus all builtins? Or the core > language, builtins and the standard library? Because I think that the > vast majority of users would expect a core/stdlib function like > subinterpreters to support the full core+stdlib language. Agreed. > So my question would be, do all of the stdlib C extension modules > support subinterpreters[1]? If they don't, then I think it's very > reasonable to expect that to be fixed, in the spirit of "eating our > own dogfood" - if we aren't willing or able to make the stdlib support > subinterpreters, it's not exactly reasonable or fair to expect 3rd > party extensions to do so. That is definitely the right question. :) Honestly I had not thought of it that way (nor checked of course). While many stdlib modules have been updated to use heap types (see PEP 384) and support PEP 489 (Multi-phase Extension Module Initialization), there are still a few stragglers. Furthermore, I expect that there are few modules that would give us trouble (maybe ssl, cdecimal). It's all about global state that gets shared inadvertently between subinterpreters. Probably the best way to find out is to run the entire test suite in a subinterpreter. I'll do that as soon as I can. > If, on the other hand, the stdlib *is* supported, then I think that > "all of Python and the stdlib, plus all 3rd party pure Python > packages" is a significant base of functionality, and an entirely > reasonable starting point for the feature. Yep, that's what I meant. I just need to identify modules where we need fixes. Thanks for bringing this up! > It certainly still excludes > big parts of the Python ecosystem (notably scientific / data science > users) but that seems fine to me - big extension users like those can > be expected to have additional limitations. It's not really that > different from the situation around C extension support in PyPy. Agreed. -eric ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/TGW2OESYUGMMRVU6JIXQXWEP3VMH7WPL/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
On Wed, Apr 29, 2020 at 5:40 AM Julien Salort wrote: > Le 29/04/2020 à 03:18, Eric Snow a écrit : > > > My (honest) question is, how many folks using subinterpreters are > > going to want to use numpy (or module X) enough to get mad about it > > before the extension supports subinterpreters? What will user > > expectations be when it comes to subinterpreters? > > > > We will make the docs as clear as we can, but there are plenty of > > users out there that will not pay enough attention to know that most > > extension modules will not support subinterpreters at first. Is there > > anything we can do to mitigate this impact? How much would it help if > > the ImportError for incompatible modules give a clear (though > > lengthier) explanation of the situation? > > For what it's worth, I can give you the feedback of a simple user. It > happens that I tried some time ago to use Numpy in a Flask project which > was deployed with mod_wsgi on an Apache server. Basically, the page was > dynamically generating some plots. And I got weird unreliable behaviour, > which took me some time to debug. > > I had to look it up on the internet to figure out the problem was that > Numpy cannot reliably work with mod_wsgi. I originally thought that I > had made a mistake somewhere in my code instead. So, I rewrote the code > to remove the dependency on Numpy. I had used Numpy in the first place, > because, as a physicist, this is what I am used to, but it was clearly > very possible to rewrite this particular code without Numpy. > > If your proposal leads to an intelligible actual error, and a clear > warning in the documentation, instead of a silent crash, this sounds > like progress, even for those packages which won't work on > subinterpreters anytime soon... > > +10 to this, the mysterious failures of today are way worse than a clear "this module doesn't support this execution environment" ImportError. I'm not worried at all about someone going and filing an issue in a project saying "please support this execution environment". Someone may eventually come along and decide they want to be the one to make that happen and do the work because they have a reason. -gps ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/YMHYG7B6U3ACMOSB3A3A7VJXQA5AMJL5/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
On Tue, 2020-04-28 at 19:20 -0600, Eric Snow wrote: > On Tue, Apr 21, 2020 at 11:17 AM Sebastian Berg > wrote: > > Maybe one of the frustrating points about this criticism is that it > > does not belong in this PEP. And that is actually true! I > > wholeheartedly agree that it doesn't really belong in this PEP > > itself. > > > > *But* the existence of a document detailing the "state and vision > > for > > subinterpreters" that includes these points is probably a > > prerequisite > > for this PEP. And this document must be linked prominently from the > > PEP. > > > > So the suggestion should maybe not be to discuss it in the PEP, but > > to > > to write it either in the documentation on subinterpreters or as an > > informational PEP. Maybe such document already exists, but then it > > is > > not linked prominently enough probably. > > That is an excellent point. It would definitely help to have more > clarity about the feature (subinterpreters). I'll look into what > makes the most sense. I've sure Victor has already effectively > written something like this. :) > I will note one more time that I want to back up almost all that Nathaniel said (I simply cannot judge the technical side though). While I still think it is probably not part of PEP 554 as such, I guess it needs a full blown PEP on its own. Saying that Python should implement subinterpreters. (I am saying "implement" because I believe you must consider subinterpreters basically a non-feature at this time. It has neither users nor reasonable ecosystem support.) In many ways I assume that a lot of the ground work for subinterpreters was useful on its own. But please do not underestimate how much effort it will take to make subinterpreters first class citizen in the language! Take PyPy for example, it took years for PyPy support in NumPy (and the PyPy people did pretty much _all_ the work). And PyPy provides a compatibility layer that makes the support orders of magnitude simpler than supporting subinterpreters. And yet, I am sure there are many many C-Extensions out there that will fail on PyPy. So unless the potential subinterpreter userbase is magnitudes larger than PyPy's the situation will be much worse. With the added frustration because PyPy users probably expect incompatibilities, but Python users may get angry if they think subinterpreters are a language feature. There have been points made about e.g. just erroring on import for modules which do not choose to support subinterpreters. And maybe in the sum of saying: * We warn that most C-extensions won't work -> If you error, at least it won't crash silently (some mitigation) * Nobody must expect any C-extension to work until subinterpreters have proven useful *and* a large userbase! * In times, we are taking here about, what? - Maybe 3 years until proven useful and a potentially large userbase? - Some uncertain amount longer until the user-base actually grows - Maybe 5 years until fairly widespread support for some central libraries after that? (I am sure Python 2 to 3 took that long) * Prototyping the first few years (such as an external package, or even a fork!) are not really very good, because... ? Or alternatively the warnings will be so penetrant that prototyping within cpython is acceptable. Maybe you have to use: python --subinterpreters-i-know-this-is-only-a-potential-feature-which-may-be-removed-in-future-python-versions myscript.py This is not in the same position as most "experimental" APIs, which are almost settled but you want to be careful. This one has a real chance of getting ripped out entirely!? is good enough, maybe it is not. Once it is written down I am confident the Python devs and steering council will make the right call. Again, I do not want to hinder the effort. It takes courage and a good champion to go down such a long and windy road. But Nathaniel is right that putting in the effort puts you into the trap of thinking that now that we are 90% there from a Python perspective, we should go 100%. 100% is nice, but you may have to reach 1000++% (i.e. C-extension modules/ecysystem support) to actually have a fully functional feature. You should get the chance to prove them useful, there seems enough positivity around the idea. But the question is within what framework that can reasonably happen. Simply pushing in PEP 554 with a small warning in the documentation is not the right framework. I hope you can find the right framework to push this on. But unfortunately it is the difficult job of the features champion. And Nathaniel, etc. are actually trying to help you with it (but in the end are not champions for it, so you cannot expect too much). E.g. by asking if it cannot be developed outside of cpython and pointing out how other similar project approached the same impassible mountain of work. Believe me, I have been there and its tough to write these documents and then get feedback which you are not
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
> On 29 Apr 2020, at 03:50, Eric Snow wrote: > > On Wed, Apr 22, 2020 at 2:43 AM Ronald Oussoren > wrote: >> My mail left out some important information, sorry about that. > > No worries. :) > >> PyObjC is a two-way bridge between Python and Objective-C. One half of this >> is that is bridging Objective-C classes (and instances) to Python. This is >> fairly straightforward, although the proxy objects are not static and can >> have methods defined in Python (helper methods that make the Objective-C >> classes nicer to use from Python, for example to define methods that make it >> possible to use an NSDictionary as if it were a regular Python dict). > > Cool. (also fairly straightforward!) Well… Except that the proxy classes are created dynamically and the list of methods is updated dynamically as well both for performance reasons and because ObjC classes can be changed at runtime (similar to how you can add methods to Python classes). But in the end this part is fairly straightforward and comparable to something like gobject introspection in the glib/gtk bindings. And every Cocoa class is proxied with a regular class and a metaclass (with parallel class hierarchies). That’s needed to mirror Objective-C behaviour, where class- and instance methods have separate namespaces and some classes have class and instance methods with the same name. > >> The other half is that it is possible to implement Objective-C classes in >> Python: >> >> class MyClass (Cocoa.NSObject): >> def anAction_(self, sender): … >> >> This defines a Python classes named “MyClass”, but also an Objective-C class >> of the same name that forwards Objective-C calls to Python. > > Even cooler! :) > >> The implementation for this uses PyGILState_Ensure, which AFAIK is not yet >> useable with sub-interpreters. > > That is correct. It is one of the few major subinterpreter > bugs/"bugs" remaining to be addressed in the CPython code. IIRC, > there were several proposed solutions (between 2 BPO issues) that > would fix it but we got distracted before the matter was settled. This is not a hard technical problem, although designing a future proof API might be harder. > >> PyObjC also has Objective-C proxy classes for generic Python objects, making >> it possible to pass a normal Python dictionary to an Objective-C API that >> expects an NSDictionary instance. > > Also super cool. How similar is this to Jython and IronPython? I don’t know, I guess this is similar to how those projects proxy between their respective host languages and Python. > >> Things get interesting when combining the two with sub-interpreters: With >> the current implementation the Objective-C world would be a channel for >> passing “live” Python objects between sub-interpreters. > > +1 > >> The translation tables for looking up existing proxies (mapping from Python >> to Objective-C and vice versa) are currently singletons. >> >> This is probably fixable with another level of administration, by keeping >> track of the sub-interpreter that owns a Python object I could ensure that >> Python objects owned by a different sub-interpreter are proxied like any >> other Objective-C object which would close this loophole. That would >> require significant changes to a code base that’s already fairly complex, >> but should be fairly straightforward. > > Do you think there are any additions we could make to the C-API (more > than have been done recently, e.g. PEP 573) that would make this > easier. From what I understand, this pattern of a cache/table of > global Python objects is a relatively common one. So anything we can > do to help transition these to per-interpreter would be broadly > beneficial. Ideally it would be done in the least intrusive way > possible, reducing churn and touch points. (e.g. a macro to convert > existing tables,etc. + an init func to call during module init.) I can probably fix this entirely on my end: - Use PEP 573 to move the translation tables to per-interpreter storage - Store the sub-interpreter in the ObjC proxy object for Python objects, both to call back into the right subinterpreter in upcalls and to tweak the way “foreign” objects are proxied into a different subinterpreter. But once again, that’s without trying to actually do the work. As usual the devil’s in the details. > > Also, FWIW, I've been thinking about possible approaches where the > first/main interpreter uses the existing static types, etc. and > further subinterpreters use a heap type (etc.) derived mostly > automatically from the static one. It's been on my mind because this > is one of the last major hurdles to clear in the CPython code before > we can make the GIL per-interpreter. > >>> What additional API would be needed? >> >> See above, the main problem is PyGILState_Ensure. I haven’t spent a lot of >> time thinking about this though, I might find other issues when I try to >> support
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
Le 29/04/2020 à 03:18, Eric Snow a écrit : My (honest) question is, how many folks using subinterpreters are going to want to use numpy (or module X) enough to get mad about it before the extension supports subinterpreters? What will user expectations be when it comes to subinterpreters? We will make the docs as clear as we can, but there are plenty of users out there that will not pay enough attention to know that most extension modules will not support subinterpreters at first. Is there anything we can do to mitigate this impact? How much would it help if the ImportError for incompatible modules give a clear (though lengthier) explanation of the situation? For what it's worth, I can give you the feedback of a simple user. It happens that I tried some time ago to use Numpy in a Flask project which was deployed with mod_wsgi on an Apache server. Basically, the page was dynamically generating some plots. And I got weird unreliable behaviour, which took me some time to debug. I had to look it up on the internet to figure out the problem was that Numpy cannot reliably work with mod_wsgi. I originally thought that I had made a mistake somewhere in my code instead. So, I rewrote the code to remove the dependency on Numpy. I had used Numpy in the first place, because, as a physicist, this is what I am used to, but it was clearly very possible to rewrite this particular code without Numpy. If your proposal leads to an intelligible actual error, and a clear warning in the documentation, instead of a silent crash, this sounds like progress, even for those packages which won't work on subinterpreters anytime soon... Cheers, Julien ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/OZQGOK5AI7BG7TJGVEJ4PPCQXZSQ4VGW/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
On Wed, 29 Apr 2020 at 02:26, Eric Snow wrote: > Subinterpreters run all Python code right now. I'm guessing by > "general python code" you are talking about the code folks are writing > plus their dependencies. In that case, it's only with extension > modules that we run into a problem, and we still don't know with how > many of those it's a problem where it will take a lot of work. > However, I *am* convinced that there is a non-trivial amount of work > there and that it impacts large extension modules more than others. > The question is, what can we do to mitigate the amount of work there? One thing that isn't at all clear to me here is that when you say "Subinterpreters run all Python code", do you *just* mean the core language? Or the core language plus all builtins? Or the core language, builtins and the standard library? Because I think that the vast majority of users would expect a core/stdlib function like subinterpreters to support the full core+stdlib language. So my question would be, do all of the stdlib C extension modules support subinterpreters[1]? If they don't, then I think it's very reasonable to expect that to be fixed, in the spirit of "eating our own dogfood" - if we aren't willing or able to make the stdlib support subinterpreters, it's not exactly reasonable or fair to expect 3rd party extensions to do so. If, on the other hand, the stdlib *is* supported, then I think that "all of Python and the stdlib, plus all 3rd party pure Python packages" is a significant base of functionality, and an entirely reasonable starting point for the feature. It certainly still excludes big parts of the Python ecosystem (notably scientific / data science users) but that seems fine to me - big extension users like those can be expected to have additional limitations. It's not really that different from the situation around C extension support in PyPy. Paul [1] Calling threading from a subinterpreter would be an interesting test of that ;-) ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/XORJE2ECDGIVIWMVME3RE4O77YGETQ6G/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
On Mon, Apr 20, 2020 at 6:21 PM Eric Snow wrote: > > Nathaniel, > > Your tone and approach to this conversation concern me. I appreciate > that you have strong feelings here and readily recognize I have my own > biases, but it's becoming increasingly hard to draw any constructive > insight from what tend to be very longs posts from you. It ends up > being a large commitment of time for small gains. And honestly, it's > also becoming hard to not counter some of your more elaborate > statements with my own unhelpful prose. In the interest of making > things better, please take it all down a notch or two. I'm sorry it's landing that way on you. I am frustrated, and I think that's a reasonable reaction. But I know we're all here because we want to make Python better. So let me try again to explain my position, to maybe reboot the conversation in a more productive way. All engineering decisions come down to costs vs. benefits. My frustration is about how you're approaching the costs, and how you're approaching the benefits. **Costs** I think you've been downplaying the impact of subinterpreter support on the existing extension ecosystem. All features have a cost, which is why PEPs always require substantial rationales and undergo intense scrutiny. But subinterpreters are especially expensive. Most features only affect a small group of modules (e.g. async/await affected twisted and tornado, but 99% of existing libraries didn't care); OTOH subinterpreters require updates to every C extension module. And if we start telling users that subinterpreters are a supported way to run arbitrary Python code, then we've effectively limited extension authors options to "update to support subinterpreters" or "explain to users why they aren't writing a proper Python module", which is an intense amount of pressure; for most features maintainers have the option of saying "well, that isn't relevant to me", but with subinterpreter support that option's been removed. (You object to my calling this an API break, but you're literally saying that old code that worked fine is being redefined to be incorrect, and that all maintainers need to learn new techniques. That's the definition of an API break!) And until everything is updated, you're creating a schism in the ecosystem, between modules that support subinterpreters and those that don't. I did just read your reply to Sebastian, and it sounds like you're starting to appreciate this impact more, which I'm glad to see. None of this means that subinterpreters are necessarily a bad idea. For example, the Python 2 -> Python 3 transition was very similar, in terms of maintainers being forced to go along and creating a temporary schism in the ecosystem, and that was justified by the deep, unfixable problems with Python 2. But it does mean that subinterpreters need an even stronger rationale than most features. And IMO, the point where PEP 554 is accepted and we start adding new public APIs for subinterpreters is the point where most of these costs kick in, because that's when we start sending the message that this is a real thing and start forcing third-party maintainers to update their code. So that's when we need the rationale. **Benefits** In talks and informal conversations, you paint a beautiful picture of all the wonderful things subinterpreters will do. Lots of people are excited by these wonderful things. I tried really hard to be excited too. (In fact I spent a few weeks trying to work out a subinterpreter-style proposal myself way back before you started working on this!) But the problem is, whenever I look more closely at the exciting benefits, I end up convincing myself that they're a mirage, and either they don't work at all (e.g. quickly sharing arbitrary objects between interpreters), or else end up being effectively a more complex, fragile version of things that already exist. I've been in lots of groups before where everyone (including me!) got excited about a cool plan, focused exclusively on the positives, and ignored critical flaws until it was too late. See also: "groupthink", "confirmation bias", etc. The whole subinterpreter discussion feels very familiar that way. I'm worried that that's what's happening. Now, I might be right, or I might be wrong, I dunno; subinterpreters are a complex topic. Generally the way we sort these things out is to write down the arguments for and against and figure out the technical merits. That's one of the purposes of writing a PEP. But: you've been *systematically refusing to do this.* Every time I've raised a concern about one rationale, then instead of discussing the technical substance of my concern, you switch to a different rationale, or say "oh well, that rationale isn't the important one right now". And the actual text in PEP 554 is *super* vague, like it's so vague it's kind of an insult to the PEP process. >From your responses in this thread, I think your core position now is that the rationale is irrelevant,
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
On Sun, Apr 26, 2020 at 2:21 PM Carl Shapiro wrote: > While this PEP may not create a maintenance burden for CPython, it > does have the effect of raising the complexity bar for an alternative > Python implementation. FWIW, I did reach out to the various major Python implementation about this and got a favorable response. See: https://www.python.org/dev/peps/pep-0554/#alternate-python-implementations > A thought that may have already been mentioned elsewhere: perhaps the > PEP could be more made more acceptable by de-scoping it to expose a > minimal set of C-API hooks to enable third-party libraries for the > sub-interpreter feature rather than providing that feature in the > standard library? Note that the low-level implementation of PEP 554 is an extension module that only uses the public C-API (no "internal" C-API). So all the hooks are already there. Or did I misunderstand? -eric ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/5GGT546APMHRFPIVZS4DW6MQ2FFFU32I/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
On Wed, Apr 22, 2020 at 2:43 AM Ronald Oussoren wrote: > My mail left out some important information, sorry about that. No worries. :) > PyObjC is a two-way bridge between Python and Objective-C. One half of this > is that is bridging Objective-C classes (and instances) to Python. This is > fairly straightforward, although the proxy objects are not static and can > have methods defined in Python (helper methods that make the Objective-C > classes nicer to use from Python, for example to define methods that make it > possible to use an NSDictionary as if it were a regular Python dict). Cool. (also fairly straightforward!) > The other half is that it is possible to implement Objective-C classes in > Python: > >class MyClass (Cocoa.NSObject): >def anAction_(self, sender): … > > This defines a Python classes named “MyClass”, but also an Objective-C class > of the same name that forwards Objective-C calls to Python. Even cooler! :) > The implementation for this uses PyGILState_Ensure, which AFAIK is not yet > useable with sub-interpreters. That is correct. It is one of the few major subinterpreter bugs/"bugs" remaining to be addressed in the CPython code. IIRC, there were several proposed solutions (between 2 BPO issues) that would fix it but we got distracted before the matter was settled. > PyObjC also has Objective-C proxy classes for generic Python objects, making > it possible to pass a normal Python dictionary to an Objective-C API that > expects an NSDictionary instance. Also super cool. How similar is this to Jython and IronPython? > Things get interesting when combining the two with sub-interpreters: With the > current implementation the Objective-C world would be a channel for passing > “live” Python objects between sub-interpreters. +1 > The translation tables for looking up existing proxies (mapping from Python > to Objective-C and vice versa) are currently singletons. > > This is probably fixable with another level of administration, by keeping > track of the sub-interpreter that owns a Python object I could ensure that > Python objects owned by a different sub-interpreter are proxied like any > other Objective-C object which would close this loophole. That would require > significant changes to a code base that’s already fairly complex, but should > be fairly straightforward. Do you think there are any additions we could make to the C-API (more than have been done recently, e.g. PEP 573) that would make this easier. From what I understand, this pattern of a cache/table of global Python objects is a relatively common one. So anything we can do to help transition these to per-interpreter would be broadly beneficial. Ideally it would be done in the least intrusive way possible, reducing churn and touch points. (e.g. a macro to convert existing tables,etc. + an init func to call during module init.) Also, FWIW, I've been thinking about possible approaches where the first/main interpreter uses the existing static types, etc. and further subinterpreters use a heap type (etc.) derived mostly automatically from the static one. It's been on my mind because this is one of the last major hurdles to clear in the CPython code before we can make the GIL per-interpreter. > > What additional API would be needed? > > See above, the main problem is PyGILState_Ensure. I haven’t spent a lot of > time thinking about this though, I might find other issues when I try to > support sub-interpreters. Any feedback on this would be useful. > >> As far as I understand proper support for subinterpreters also requires > >> moving > >> away from static type definitions to avoid sharing objects between > >> interpreters > >> (that is, use the PyType_FromSpec to build types). > > > > Correct, though that is not technically a problem until we stop sharing the > > GIL. > > Right. But a major selling point of sub-interpreters is that this provide a > way forward towards having multiple Python threads that don’t share a GIL. > > IMHO it would be better to first work out what’s needed to get there, and in > particular what changes are needed in extensions. Otherwise extensions may > have to be changed multiple times. Yeah, I see what you're saying. It's been a hard balance to strike. There are really 2 things that have to be done: move all global state to per-interpreter and deal with static types (etc.). Do you think both will require significant work in the community? My instinct, partly informed by my work in CPython along these lines, is that the former is more work and sometimes trickier. The latter is fairly straightforward and much more of an opportunity for automatic approaches. > >> At first glance this API does not support everything I do in PyObjC (fun > >> with metaclasses, in C code). > > > > What specific additions/changes would you need? > > At least: > > - A variant of PyGILState_Ensure that supports sub-interpreters > - Defining subclasses of
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
On Tue, Apr 21, 2020 at 11:17 AM Sebastian Berg wrote: > Maybe one of the frustrating points about this criticism is that it > does not belong in this PEP. And that is actually true! I > wholeheartedly agree that it doesn't really belong in this PEP itself. > > *But* the existence of a document detailing the "state and vision for > subinterpreters" that includes these points is probably a prerequisite > for this PEP. And this document must be linked prominently from the > PEP. > > So the suggestion should maybe not be to discuss it in the PEP, but to > to write it either in the documentation on subinterpreters or as an > informational PEP. Maybe such document already exists, but then it is > not linked prominently enough probably. That is an excellent point. It would definitely help to have more clarity about the feature (subinterpreters). I'll look into what makes the most sense. I've sure Victor has already effectively written something like this. :) -eric ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/2W2GOKCUV35BAOX75ILD7XOIGWCXJ5KN/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
On Tue, Apr 21, 2020 at 10:24 AM Sebastian Berg wrote: > On Tue, 2020-04-21 at 16:21 +0200, Victor Stinner wrote: > > I fail to follow your logic. When the asyncio PEP was approved, I > > don't recall that suddenly the whole Python community started to > > rewrite all projects to use coroutines everywhere. I tried hard to > > replace eventlet with asyncio in OpenStack and I failed because such > > migration was a very large project with dubious benefits (people > > impacted by eventlet issues were the minority). > > Sure, but this is very different. You can still use NumPy in a project > using asyncio. You are _not_ able to use NumPy in a project using > subinterpreters. True. Is that a short-term problem? I don't know. A long-term problem? Definitely. So it will have to be addressed at some point. The biggest concern here is what is the resulting burden on extension authors and what can we do to help mitigate that. The first step is to understand what that burden might entail. > Right now, I have to say as soon as the first bug report asking for > this is opened and tells me: But see PEP 554 you should support it! I > would be tempted to put on the NumPy Roadmap/Vision that no current > core dev will put serious efforts into subinterpreters. Someone is > bound to be mad. Yeah. And I don't want to put folks in the position that they get fussed at for something like this. This isn't a ploy to force projects like numpy to fix their subinterpreter support. My (honest) question is, how many folks using subinterpreters are going to want to use numpy (or module X) enough to get mad about it before the extension supports subinterpreters? What will user expectations be when it comes to subinterpreters? We will make the docs as clear as we can, but there are plenty of users out there that will not pay enough attention to know that most extension modules will not support subinterpreters at first. Is there anything we can do to mitigate this impact? How much would it help if the ImportError for incompatible modules give a clear (though lengthier) explanation of the situation? > Basically, if someone wants it in NumPy, I personally may expect them > to be prepared to invest a year worth of good dev time [1]. Maybe that > is pessimistic, but your guess is as good as mine. At normal dev-pace > it will be at least a few years of incremental changes before NumPy > might be ready (how long did it take Python?)? > > The PEP links to NumPy bugs, I am not sure that we ever fixed a single > one. Even if, the remaining ones are much larger and deeper. As of now, > the NumPy public API has to be changed to even start supporting > subinterpreters as far as I aware [2]. This is because right now we > sometimes need to grab the GIL (raise errors) in functions that are not > passed GIL state. What do you expect to have to change? It might not be as bad as you think...or I suppose it could be. :) Keep in mind that subinterpreter support means making sure all of the module's global state is per-interpreter. I'm hearing about things like passing around GIL state and using the limited C-API. None of that should be a factor. > This all is not to say that this PEP itself doesn't seem harmless. But > the _expectation_ that subinterpreters should be first class citizens > will be a real and severe transition burden. And if it does not, the > current text of the PEP gives me, as someone naive about > subinterpreters, very few reasons why I should put in that effort or > reasons to make me believe that it actually is not as bad a transition > as it seems. Yeah, the PEP is very light on useful information extension module maintainers. What information do you think would be most helpful? > Right now, I would simply refuse to spend time on it. But as Nathaniel > said, it may be worse if I did not refuse and in the end only a handful > of users get anything out of my work: The time is much better spend > elsewhere. And you, i.e. CPython will spend your "please fix your C- > extension" chips on subinterpreters. Maybe that is the only thing on > the agenda, but if it is not, it could push other things away. Good point. > Reading the PEP, it is fuzzy on the promises (the most concrete I > remember is that it may be good for security relevant reasons), which > is fine, because the goal is "experimentation" more than use? The PEP is definitely lacking clear details on how folks might use subinterpreters (via the proposed module). There are a variety of reasons. I originally wrote the PEP mostly as "let's expose existing functionality more broadly", with the goal of getting it into folks' hands sooner rather than later. My focus was mostly on the API. I didn't see a strong need to convince anyone that the feature itself was worth it (since it already existed). In many ways the PEP is a side effect of my efforts to achieve a good multi-core Python story (via a per-interpreter GIL). All the relevant parties in that
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
On Tue, Apr 21, 2020 at 11:31 PM Greg Ewing wrote: > To put it another way, the moment you start using subinterpreters, > the set of extension modules you are able to use will shrink > *enormously*. Very true but we have to start somewhere. > And if I understand correctly, you won't get any nice "This > module does not support subinterpreters" exception if you > import an incompatible module -- just an obscure crash, > probably of the core-dumping variety. As Petr noted, we can use PEP 489 (Multi-phase Extension Module Initialization) support as an indicator and raise ImportError for any other extension modules (when in a subinterpreter). That seems like a reasonable way to avoid the hard-to-debug failures that would result otherwise. The only question I have is if it makes sense to offer a way to disable such a check (e.g. a flag when creating a subinterpreter). I think so, since then extension authors could more easily test their extension under subinterpreters without having to release a separate build that has PEP 489 support. -eric ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/IB5IAQUMPFV3W2V3EBBKWRJWROZ42OVD/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
On Tue, Apr 21, 2020 at 7:33 AM Victor Stinner wrote: > IMHO it's worth it to explore the subinterpreters approach. I don't > think that it's going to be a major maintenance burden: the code is > already written and tests. The PEP is only about adding a public > Python API. While this PEP may not create a maintenance burden for CPython, it does have the effect of raising the complexity bar for an alternative Python implementation. > Even today, asyncio didn't replace threads, multiprocessing, > concurrent.futures, etc. There are even competitor projects like > Twisted, trio and curio! (Also eventlet and gevent based on greenlet > which is a different approach). I only started to see very recently > project like httpx which supports both blocking and asynchronous API. Because asyncio had been implemented as a library, the up-take of asyncio could have been lower because the demand was fulfilled, at least in part, by those third-party libraries? A thought that may have already been mentioned elsewhere: perhaps the PEP could be more made more acceptable by de-scoping it to expose a minimal set of C-API hooks to enable third-party libraries for the sub-interpreter feature rather than providing that feature in the standard library? ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/KSRMK2OULUHM4KTKAFXWO6RFM2F2IWE6/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
On 2020-04-22 08:05, Glenn Linderman wrote: On 4/21/2020 10:26 PM, Greg Ewing wrote: And if I understand correctly, you won't get any nice "This module does not support subinterpreters" exception if you import an incompatible module -- just an obscure crash, probably of the core-dumping variety. This sounds fixable: modules that support subinterpreters should set a flag saying so, and the either the load of a non-flagged module when subinterpreters are in use, or the initiation of a subinterpreter when a non-flagged module has been loaded, should raise. There was talk about making a orthogonal flag just for opting into subinterpreter support. But what you use now as such a flag is multi-phase initialization: https://docs.python.org/3/c-api/module.html?highlight=pymodule_fromdefandspec#multi-phase-initialization (Though I like the PEP 489 wording, "expected to support subinterpreters and multiple Py_Initialize/Py_Finalize cycles correctly", better than what ended up in the docs.) The simplest strategy to support subinterpreters correctly is to refuse to create the extension module more than once per process (with an appropriate error, of course). ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/5J5XTTDUKZWEVDSU3V67W3FYOY7VA7UO/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
> On 21 Apr 2020, at 16:58, Eric Snow wrote: > > Thanks for explaining that, Ronald. It sounds like a lot of the > effort would relate to making classes work. I have some comments > in-line below. > > -eric > > On Tue, Apr 21, 2020 at 2:34 AM Ronald Oussoren > wrote: >>> On 21 Apr 2020, at 03:21, Eric Snow wrote: >>> Honest question: how many C extensions have process-global state that >>> will cause problems under subinterpreters? In other words, how many >>> already break in mod_wsgi? >> >> Fully supporting sub-interpreters in PyObjC will likely be a lot of work, >> mostly >> due to being able to subclass Objective-C classes from Python. With sub- >> interpreters a Python script in an interpreter could see an Objective-C >> class in >> a different sub-interpreter. The current PyObjC architecture assumes that >> there’s exactly one (sub-)interpreter, that’s probably fixable but is far >> from trivial. > > Are the Objective-C classes immutable? Are the wrappers stateful at > all? > Without context I'm not clear on how you would be impacted by > operation under subinterpreters (i.e. PEP 554), but it sounds like the > classes do have global state that is in fact interpreter-specific. I > expect you would also be impacted by subinterpreters not sharing the > GIL but that is a separate matter (see below). My mail left out some important information, sorry about that. PyObjC is a two-way bridge between Python and Objective-C. One half of this is that is bridging Objective-C classes (and instances) to Python. This is fairly straightforward, although the proxy objects are not static and can have methods defined in Python (helper methods that make the Objective-C classes nicer to use from Python, for example to define methods that make it possible to use an NSDictionary as if it were a regular Python dict). The other half is that it is possible to implement Objective-C classes in Python: class MyClass (Cocoa.NSObject): def anAction_(self, sender): … This defines a Python classes named “MyClass”, but also an Objective-C class of the same name that forwards Objective-C calls to Python. The implementation for this uses PyGILState_Ensure, which AFAIK is not yet useable with sub-interpreters. PyObjC also has Objective-C proxy classes for generic Python objects, making it possible to pass a normal Python dictionary to an Objective-C API that expects an NSDictionary instance. Things get interesting when combining the two with sub-interpreters: With the current implementation the Objective-C world would be a channel for passing “live” Python objects between sub-interpreters. The translation tables for looking up existing proxies (mapping from Python to Objective-C and vice versa) are currently singletons. This is probably fixable with another level of administration, by keeping track of the sub-interpreter that owns a Python object I could ensure that Python objects owned by a different sub-interpreter are proxied like any other Objective-C object which would close this loophole. That would require significant changes to a code base that’s already fairly complex, but should be fairly straightforward. > > Regardless, I expect there are others in a similar situation. It > would be good to understand your use case and help with a solution. > Is there a specific example you can point to of code that would be > problematic under subinterpreters? > >> With the current API it might not even be possible to add sub-interpreter >> support > > What additional API would be needed? See above, the main problem is PyGILState_Ensure. I haven’t spent a lot of time thinking about this though, I might find other issues when I try to support sub-interpreters. > >> (although I write this without having read the PEP). > > Currently PEP 554 does not talk about how to make extension modules > compatible with subinterpreters. That may be worth doing, though it > would definitely have to happen in the docs and (to an extent) the 3.9 > "What's New" page. There is already some discussion on what should be > in those docs (see > https://github.com/ericsnowcurrently/multi-core-python/issues/53). > > Note that, until the GIL becomes per-interpreter, sharing objects > isn't a problem. We were not planning on having a PEP for the > stop-sharing-the-GIL effort, but I'm starting to think that it may be > worth it, to cover the impact on extension modules (e.g. mitigations). > > So if you leave out the complications due to not sharing the GIL, the > main problem extension authors face with subinterpreters (exposed by > PEP 554) is when their module has process-global state that breaks > under subinterpreters. From your description above, it sounds like > you may be in that situation. > >> As far as I understand proper support for subinterpreters also requires >> moving >> away from static type definitions to avoid sharing objects between >> interpreters >> (that is, use
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
On 4/21/2020 10:26 PM, Greg Ewing wrote: And if I understand correctly, you won't get any nice "This module does not support subinterpreters" exception if you import an incompatible module -- just an obscure crash, probably of the core-dumping variety. This sounds fixable: modules that support subinterpreters should set a flag saying so, and the either the load of a non-flagged module when subinterpreters are in use, or the initiation of a subinterpreter when a non-flagged module has been loaded, should raise. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/H3TCU43YUSUMYUMWBA6PFZZJ7AG5FZCD/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
On 22/04/20 4:21 am, Sebastian Berg wrote: Sure, but this is very different. You can still use NumPy in a project using asyncio. You are _not_ able to use NumPy in a project using subinterpreters. To put it another way, the moment you start using subinterpreters, the set of extension modules you are able to use will shrink *enormously*. And if I understand correctly, you won't get any nice "This module does not support subinterpreters" exception if you import an incompatible module -- just an obscure crash, probably of the core-dumping variety. To me this would feel like programming in the middle of a lethal minefield. -- Greg ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/OLTRZWS5SRBXXLB25C44DE5MTTFZDWJS/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
On Tue, 21 Apr 2020 12:05:28 -0700 "Gregory P. Smith" wrote: > On Tue, Apr 21, 2020 at 10:49 AM Antoine Pitrou wrote: > > > On Tue, 21 Apr 2020 18:46:04 +0200 > > Petr Viktorin wrote: > > > On 2020-04-21 11:01, Antoine Pitrou wrote: > > > > On Mon, 20 Apr 2020 19:21:21 -0600 > > > > Eric Snow wrote: > > > >> Honest question: how many C extensions have process-global state that > > > >> will cause problems under subinterpreters? In other words, how many > > > >> already break in mod_wsgi? > > > > > > > > A slightly tricky question is what happens if a PyObject survives > > > > longer than the subinterpreter that created it. > > > > > > > > For example, in PyArrow, we allow passing a Python buffer-like object > > > > as a C++ buffer to C++ APIs. The C++ buffer could conceivably be kept > > > > around by C++ code for some time. When the C++ buffer is destroyed, > > > > Py_DECREF() is called on the Python object (I figure that we would have > > > > to switch to the future interpreter-aware PyGILState API -- when will > > > > it be available?). > > > > In this scenario given that the PyObject owning the buffer passed to C++ by > PyArrow was Py_INCREF'd, an option during Py_Finalize is to not release > anything who's refcount never made it to 0. Which also implies leaving > enough of the interpreter state around so that Py_DECREF could be called > and trigger the free(). That's assuming a trivial destructor. Of course, that would cater for most concrete use cases. Regards Antoine. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/CJFM5WBERJU4NLW6QJ7VISGFVBCQ3HMA/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
On Tue, Apr 21, 2020 at 10:49 AM Antoine Pitrou wrote: > On Tue, 21 Apr 2020 18:46:04 +0200 > Petr Viktorin wrote: > > On 2020-04-21 11:01, Antoine Pitrou wrote: > > > On Mon, 20 Apr 2020 19:21:21 -0600 > > > Eric Snow wrote: > > >> Honest question: how many C extensions have process-global state that > > >> will cause problems under subinterpreters? In other words, how many > > >> already break in mod_wsgi? > > > > > > A slightly tricky question is what happens if a PyObject survives > > > longer than the subinterpreter that created it. > > > > > > For example, in PyArrow, we allow passing a Python buffer-like object > > > as a C++ buffer to C++ APIs. The C++ buffer could conceivably be kept > > > around by C++ code for some time. When the C++ buffer is destroyed, > > > Py_DECREF() is called on the Python object (I figure that we would have > > > to switch to the future interpreter-aware PyGILState API -- when will > > > it be available?). > In this scenario given that the PyObject owning the buffer passed to C++ by PyArrow was Py_INCREF'd, an option during Py_Finalize is to not release anything who's refcount never made it to 0. Which also implies leaving enough of the interpreter state around so that Py_DECREF could be called and trigger the free(). Basically (sub)interpreter finalization should have the right to return failure. Everything ever allocated by an interpreter holds a reference (implied today) to its associated interpreter state. -gps > > > > > > > But what happens if the interpreter which created > > > the Python object is already destroyed at this point? > > > > That's a good question. What happens today if someone calls Py_Finalize > > while the buffer is still around? > > > > I don't think you need to treat *sub*interpreters specially. > > The difference is that subinterpreters may have a shorter lifetime than > the main interpreter. Also, we're currently using a global memory > allocator, so simple operations may work until the process dies. > Regards > > Antoine. > > ___ > Python-Dev mailing list -- python-dev@python.org > To unsubscribe send an email to python-dev-le...@python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/UWWO3RS3UOYCZXVFORGTJAEXXOCUQQXD/ > Code of Conduct: http://python.org/psf/codeofconduct/ > ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/LJJVHHK7OG6ZRISCT6G2NWBBIYDGFHVB/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
On Tue, 21 Apr 2020 18:46:04 +0200 Petr Viktorin wrote: > On 2020-04-21 11:01, Antoine Pitrou wrote: > > On Mon, 20 Apr 2020 19:21:21 -0600 > > Eric Snow wrote: > >> Honest question: how many C extensions have process-global state that > >> will cause problems under subinterpreters? In other words, how many > >> already break in mod_wsgi? > > > > A slightly tricky question is what happens if a PyObject survives > > longer than the subinterpreter that created it. > > > > For example, in PyArrow, we allow passing a Python buffer-like object > > as a C++ buffer to C++ APIs. The C++ buffer could conceivably be kept > > around by C++ code for some time. When the C++ buffer is destroyed, > > Py_DECREF() is called on the Python object (I figure that we would have > > to switch to the future interpreter-aware PyGILState API -- when will > > it be available?). > > > > But what happens if the interpreter which created > > the Python object is already destroyed at this point? > > That's a good question. What happens today if someone calls Py_Finalize > while the buffer is still around? > > I don't think you need to treat *sub*interpreters specially. The difference is that subinterpreters may have a shorter lifetime than the main interpreter. Also, we're currently using a global memory allocator, so simple operations may work until the process dies. Regards Antoine. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/UWWO3RS3UOYCZXVFORGTJAEXXOCUQQXD/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
On Tue, 2020-04-21 at 11:21 -0500, Sebastian Berg wrote: > On Tue, 2020-04-21 at 16:21 +0200, Victor Stinner wrote: > > Le mar. 21 avr. 2020 à 00:50, Nathaniel Smith a > > écrit > > : > > As far as I can tell, nobody can or _should_ expect subinterpreters > to > actually run most general python code for many years. Yes, its a > chicken-and-egg problem, unless users start to use subinterpreters > successfully, C-extensions should probably not even worry to > transition. > This PEP wants to break the chicken-and-egg problem to have a start, > but as of now, as far as I can tell, it *must not* promise that it > will > ever work out. > > So, I cannot judge the sentiment or subinterpreters. But it may be > good > to make it *painfully* clear what you expect from a project like > NumPy Maybe one of the frustrating points about this criticism is that it does not belong in this PEP. And that is actually true! I wholeheartedly agree that it doesn't really belong in this PEP itself. *But* the existence of a document detailing the "state and vision for subinterpreters" that includes these points is probably a prerequisite for this PEP. And this document must be linked prominently from the PEP. So the suggestion should maybe not be to discuss it in the PEP, but to to write it either in the documentation on subinterpreters or as an informational PEP. Maybe such document already exists, but then it is not linked prominently enough probably. - Sebastian > in the next few years. Alternatively, make it painfully clear that > you > possibly even discourage us from spending time on it now, if its not > straight forward. Those using this module are on their own for many > years, probably even after success is proven. > > Best, > > Sebastian > > > [1] As of now, the way I see it is that I could not even make NumPy > (and probably many C extensions) work, because I doubt that the > limited > API has been exercised enough [2] and I am pretty sure it has holes. > Also the PEP about passing module state around to store globals > efficiently seems necessary, and is not in yet? (Again, trust: I have > to trust you that e.g. what you do to make argument parsing not have > overhead in argument clinic will be something that I can use for > similar purposes within NumPy) > > [2] I hope that we will do (many) these changes for other reasons > within a year or so, but they go deep into code barely touched in a > decade. Realistically, even after the straight forward changes (such > as > using the new PEPs for module initialization), these may take up an > additional few months of dev time (sure, get someone very good or > does > nothing else, they can do it much quicker maybe). > So yes, from the perspective of a complex C-extension, this is > probably > very comparable to the 2to3 change (it happened largely before my > time > though). > > [3] E.g. I think I want an ExtensionMetaClass, a bit similar as an > ABC, > but I would prefer to store the data in a true C-slot fashion. The > limited API cannot do MetaClasses correctly as far as I could tell > and > IIRC is likely even a bit buggy. > Are ExtensionMetaClasses crazy? Maybe, but PySide does it too (and as > far as I can tell, they basically get away with it by a bit of > hacking > and relying on Python implementation details. > > > > > When asyncio landed in Python 3.4, a few people started to > > experiment > > it. Some had a bad experience. Some others were excited and put a > > few > > applications in production. > > > > Even today, asyncio didn't replace threads, multiprocessing, > > concurrent.futures, etc. There are even competitor projects like > > Twisted, trio and curio! (Also eventlet and gevent based on > > greenlet > > which is a different approach). I only started to see very recently > > project like httpx which supports both blocking and asynchronous > > API. > > > > I see a slow adoption of asyncio because asyncio solves very > > specific > > use cases. And that's fine! > > > > I don't expect that everyone will suddenly spend months of work to > > rewrite their C code and Python code to be more efficient or fix > > issues with subinterpreters, until a critical mass of users proved > > that subinterpreters are amazing and way more efficient! > > > > Victor > > -- > > Night gathers, and now my watch begins. It shall not end until my > > death. > > ___ > > Python-Dev mailing list -- python-dev@python.org > > To unsubscribe send an email to python-dev-le...@python.org > > https://mail.python.org/mailman3/lists/python-dev.python.org/ > > Message archived at > > https://mail.python.org/archives/list/python-dev@python.org/message/3MK2NANMOJFHKJWELGI2ZD74K2ZYJAOD/ > > Code of Conduct: http://python.org/psf/codeofconduct/ > > ___ > Python-Dev mailing list -- python-dev@python.org > To unsubscribe send an email to python-dev-le...@python.org >
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
On 2020-04-21 11:01, Antoine Pitrou wrote: On Mon, 20 Apr 2020 19:21:21 -0600 Eric Snow wrote: Honest question: how many C extensions have process-global state that will cause problems under subinterpreters? In other words, how many already break in mod_wsgi? A slightly tricky question is what happens if a PyObject survives longer than the subinterpreter that created it. For example, in PyArrow, we allow passing a Python buffer-like object as a C++ buffer to C++ APIs. The C++ buffer could conceivably be kept around by C++ code for some time. When the C++ buffer is destroyed, Py_DECREF() is called on the Python object (I figure that we would have to switch to the future interpreter-aware PyGILState API -- when will it be available?). But what happens if the interpreter which created the Python object is already destroyed at this point? That's a good question. What happens today if someone calls Py_Finalize while the buffer is still around? I don't think you need to treat *sub*interpreters specially. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/AVJO43PRTX4UCMLI2PNHMZWWBEGZS353/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
On Tue, 2020-04-21 at 16:21 +0200, Victor Stinner wrote: > Le mar. 21 avr. 2020 à 00:50, Nathaniel Smith a écrit > : > > > > tl;dr: accepting PEP 554 is effectively a C API break, and will > > force > > many thousands of people worldwide to spend many hours wrangling > > with > > subinterpreter support. > > I fail to follow your logic. When the asyncio PEP was approved, I > don't recall that suddenly the whole Python community started to > rewrite all projects to use coroutines everywhere. I tried hard to > replace eventlet with asyncio in OpenStack and I failed because such > migration was a very large project with dubious benefits (people > impacted by eventlet issues were the minority). Sure, but this is very different. You can still use NumPy in a project using asyncio. You are _not_ able to use NumPy in a project using subinterpreters. Right now, I have to say as soon as the first bug report asking for this is opened and tells me: But see PEP 554 you should support it! I would be tempted to put on the NumPy Roadmap/Vision that no current core dev will put serious efforts into subinterpreters. Someone is bound to be mad. Basically, if someone wants it in NumPy, I personally may expect them to be prepared to invest a year worth of good dev time [1]. Maybe that is pessimistic, but your guess is as good as mine. At normal dev-pace it will be at least a few years of incremental changes before NumPy might be ready (how long did it take Python?)? The PEP links to NumPy bugs, I am not sure that we ever fixed a single one. Even if, the remaining ones are much larger and deeper. As of now, the NumPy public API has to be changed to even start supporting subinterpreters as far as I aware [2]. This is because right now we sometimes need to grab the GIL (raise errors) in functions that are not passed GIL state. This all is not to say that this PEP itself doesn't seem harmless. But the _expectation_ that subinterpreters should be first class citizens will be a real and severe transition burden. And if it does not, the current text of the PEP gives me, as someone naive about subinterpreters, very few reasons why I should put in that effort or reasons to make me believe that it actually is not as bad a transition as it seems. Right now, I would simply refuse to spend time on it. But as Nathaniel said, it may be worse if I did not refuse and in the end only a handful of users get anything out of my work: The time is much better spend elsewhere. And you, i.e. CPython will spend your "please fix your C- extension" chips on subinterpreters. Maybe that is the only thing on the agenda, but if it is not, it could push other things away. Reading the PEP, it is fuzzy on the promises (the most concrete I remember is that it may be good for security relevant reasons), which is fine, because the goal is "experimentation" more than use? So if its more about "experimentation", then I have to ask, whether: 1. The PEP can state that more obviously, it wants to be provisionally/experimentally accept? So maybe it should even say that that extension modules are not (really) encouraged to transition unless they feel a significant portion of their users will gain. 2. The point about developing it outside of the Python standard lib should be considered more seriously. I do not know if that can be done, but C-API additions/changes/tweaks seem a bit orthogonal to the python exposure? So maybe it actually is possible? As far as I can tell, nobody can or _should_ expect subinterpreters to actually run most general python code for many years. Yes, its a chicken-and-egg problem, unless users start to use subinterpreters successfully, C-extensions should probably not even worry to transition. This PEP wants to break the chicken-and-egg problem to have a start, but as of now, as far as I can tell, it *must not* promise that it will ever work out. So, I cannot judge the sentiment or subinterpreters. But it may be good to make it *painfully* clear what you expect from a project like NumPy in the next few years. Alternatively, make it painfully clear that you possibly even discourage us from spending time on it now, if its not straight forward. Those using this module are on their own for many years, probably even after success is proven. Best, Sebastian [1] As of now, the way I see it is that I could not even make NumPy (and probably many C extensions) work, because I doubt that the limited API has been exercised enough [2] and I am pretty sure it has holes. Also the PEP about passing module state around to store globals efficiently seems necessary, and is not in yet? (Again, trust: I have to trust you that e.g. what you do to make argument parsing not have overhead in argument clinic will be something that I can use for similar purposes within NumPy) [2] I hope that we will do (many) these changes for other reasons within a year or so, but they go deep into code barely touched in a decade. Realistically, even after the
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
On Tue, Apr 21, 2020 at 8:54 AM Paul Moore wrote: > > On Tue, 21 Apr 2020 at 15:31, Eric Snow wrote: > > Here are the options for handling non-compliant extension modules: > > > > 1. do nothing (extensions may break in ways that are hard for users to > > deal with) > > 2. raise ImportError if an extension without PEP 489 support is > > imported in a subinterpreter > > 3. add a per-interpreter isolated mode (default) that provides the > > behavior of #2, where non-isolated gives #1 > > > > At this point my plan was go with #2, to avoid giving extension > > authors the pain that comes with #1. If we went with #3, at least > > during the provisional phase, it would allow extension authors to try > > out their extensions in subinterpreters. > > This seems sensible to me - #2 ensures that we don't get weird bugs > from users trying things that we know they shouldn't be doing. I'm not > sure how much advantage #3 would have, presumably extension authors > wanting to "try out" subinterpreters would at some point have to mark > them as importable as per #2, so why not just do that much and then > see what else needs changing? Good point. I suppose the main difference is that they (and their users) could try the extension under subinterpreters without needing a separate build. > But I don't know the C API for > subinterpreters, so maybe there are complexities I don't understand, > in which case having #3 probably makes more sense in that context. There aren't any extra complexities. Per PEP 489 [1], in a module's init function you call PyModuleDef_Init() on its PyModuleDef and the return it. The import machinery does the rest. The subinterpreter C-API is not involved. -eric [1] https://www.python.org/dev/peps/pep-0489/ ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/ACYSACREHESNOUIJLXZLRZUBORZ4Q7GS/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
On Tue, Apr 21, 2020 at 4:11 AM Greg Ewing wrote: > On 21/04/20 8:34 pm, Ronald Oussoren via Python-Dev wrote: > > As far as I understand proper support for subinterpreters also requires > > moving away from static type definition to avoid sharing objects > between > > interpreters (that is, use the PyType_FromSpec to build types). > > Which means updating every extension module that defines its > own Python types and was written before PyType_FromSpec existed. > I expect there is a huge number of those. Correct. Before we make the GIL per-interpreter, we will need to find a solution for things like static types which minimizes the impact on extension authors. Note that PyType_FromSpec does not have to be involved. It is part of the Stable ABI effort in PEP 384, though it does help with subinterpreters. I expect there are other solutions which will not involve as much churn, though I know folks who would want everyone to use heap types regardless. :) -eric ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/6PWGTPZIFSHVACVTZ6YFA7MYY7L22G4O/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
On Tue, Apr 21, 2020 at 3:10 AM Antoine Pitrou wrote: > On Mon, 20 Apr 2020 19:21:21 -0600 > Eric Snow wrote: > > Honest question: how many C extensions have process-global state that > > will cause problems under subinterpreters? In other words, how many > > already break in mod_wsgi? > > A slightly tricky question is what happens if a PyObject survives > longer than the subinterpreter that created it. > > For example, in PyArrow, we allow passing a Python buffer-like object > as a C++ buffer to C++ APIs. The C++ buffer could conceivably be kept > around by C++ code for some time. When the C++ buffer is destroyed, > Py_DECREF() is called on the Python object (I figure that we would have > to switch to the future interpreter-aware PyGILState API -- when will > it be available?). But what happens if the interpreter which created > the Python object is already destroyed at this point? Good question (and example). I'm not familiar enough with the GC machinery to have a good answer. I do know that GC is per-interpreter now, but the allocators are still global. I'm not sure how object lifetime is *currently* tied to interpreter lifetime. I'd expect that, at the point we have per-interpreter allocators, that all objects in that interpreter would be destroyed when the interpreter is destroyed. -eric ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/7DJCPVP7UA66WMEF7AWKAEWUC5N37WZN/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
Thanks for explaining that, Ronald. It sounds like a lot of the effort would relate to making classes work. I have some comments in-line below. -eric On Tue, Apr 21, 2020 at 2:34 AM Ronald Oussoren wrote: > > On 21 Apr 2020, at 03:21, Eric Snow wrote: > > Honest question: how many C extensions have process-global state that > > will cause problems under subinterpreters? In other words, how many > > already break in mod_wsgi? > > Fully supporting sub-interpreters in PyObjC will likely be a lot of work, > mostly > due to being able to subclass Objective-C classes from Python. With sub- > interpreters a Python script in an interpreter could see an Objective-C class > in > a different sub-interpreter. The current PyObjC architecture assumes that > there’s exactly one (sub-)interpreter, that’s probably fixable but is far > from trivial. Are the Objective-C classes immutable? Are the wrappers stateful at all? Without context I'm not clear on how you would be impacted by operation under subinterpreters (i.e. PEP 554), but it sounds like the classes do have global state that is in fact interpreter-specific. I expect you would also be impacted by subinterpreters not sharing the GIL but that is a separate matter (see below). Regardless, I expect there are others in a similar situation. It would be good to understand your use case and help with a solution. Is there a specific example you can point to of code that would be problematic under subinterpreters? > With the current API it might not even be possible to add sub-interpreter > support What additional API would be needed? > (although I write this without having read the PEP). Currently PEP 554 does not talk about how to make extension modules compatible with subinterpreters. That may be worth doing, though it would definitely have to happen in the docs and (to an extent) the 3.9 "What's New" page. There is already some discussion on what should be in those docs (see https://github.com/ericsnowcurrently/multi-core-python/issues/53). Note that, until the GIL becomes per-interpreter, sharing objects isn't a problem. We were not planning on having a PEP for the stop-sharing-the-GIL effort, but I'm starting to think that it may be worth it, to cover the impact on extension modules (e.g. mitigations). So if you leave out the complications due to not sharing the GIL, the main problem extension authors face with subinterpreters (exposed by PEP 554) is when their module has process-global state that breaks under subinterpreters. From your description above, it sounds like you may be in that situation. > As far as I understand proper support for subinterpreters also requires moving > away from static type definitions to avoid sharing objects between > interpreters > (that is, use the PyType_FromSpec to build types). Correct, though that is not technically a problem until we stop sharing the GIL. > At first glance this API does not support everything I do in PyObjC (fun with > metaclasses, in C code). What specific additions/changes would you need? ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/UIX7FM2UJUL2UIIBNOBYNP42O6RCSJME/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
On Tue, 21 Apr 2020 at 15:31, Eric Snow wrote: > Here are the options for handling non-compliant extension modules: > > 1. do nothing (extensions may break in ways that are hard for users to > deal with) > 2. raise ImportError if an extension without PEP 489 support is > imported in a subinterpreter > 3. add a per-interpreter isolated mode (default) that provides the > behavior of #2, where non-isolated gives #1 > > At this point my plan was go with #2, to avoid giving extension > authors the pain that comes with #1. If we went with #3, at least > during the provisional phase, it would allow extension authors to try > out their extensions in subinterpreters. This seems sensible to me - #2 ensures that we don't get weird bugs from users trying things that we know they shouldn't be doing. I'm not sure how much advantage #3 would have, presumably extension authors wanting to "try out" subinterpreters would at some point have to mark them as importable as per #2, so why not just do that much and then see what else needs changing? But I don't know the C API for subinterpreters, so maybe there are complexities I don't understand, in which case having #3 probably makes more sense in that context. Paul ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/KFOJQ3Z7AS3PCCAIOGA2KPGXXJWEYIZK/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
Le mar. 21 avr. 2020 à 00:50, Nathaniel Smith a écrit : > Why do you believe that subinterpreters will have reduced resource > usage? I assume you're comparing them to subprocesses here. > Subinterpreters are "shared-nothing"; all code, data, etc. has to be > duplicated, except for static C code ... which is exactly the same as > how subprocesses work. So I don't see any theoretical reason why they > should have reduced resource usage. As you, I would like to see benchmarks of subinterpreters running in parallel before saying that "yeah, that approach works to speed up this specific use case". But I don't think that it should hold the PEP 554. Eric is transparent about his intent and limitations of the current implementation. There is no silver bullet for parallelism and concurrency. For example, multiprocessing uses threads in addition to processes, and asyncio also uses threads internally to execute blocking code like DNS resolution. IMHO it's worth it to explore the subinterpreters approach. I don't think that it's going to be a major maintenance burden: the code is already written and tests. The PEP is only about adding a public Python API. > tl;dr: accepting PEP 554 is effectively a C API break, and will force > many thousands of people worldwide to spend many hours wrangling with > subinterpreter support. I fail to follow your logic. When the asyncio PEP was approved, I don't recall that suddenly the whole Python community started to rewrite all projects to use coroutines everywhere. I tried hard to replace eventlet with asyncio in OpenStack and I failed because such migration was a very large project with dubious benefits (people impacted by eventlet issues were the minority). When asyncio landed in Python 3.4, a few people started to experiment it. Some had a bad experience. Some others were excited and put a few applications in production. Even today, asyncio didn't replace threads, multiprocessing, concurrent.futures, etc. There are even competitor projects like Twisted, trio and curio! (Also eventlet and gevent based on greenlet which is a different approach). I only started to see very recently project like httpx which supports both blocking and asynchronous API. I see a slow adoption of asyncio because asyncio solves very specific use cases. And that's fine! I don't expect that everyone will suddenly spend months of work to rewrite their C code and Python code to be more efficient or fix issues with subinterpreters, until a critical mass of users proved that subinterpreters are amazing and way more efficient! Victor -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/3MK2NANMOJFHKJWELGI2ZD74K2ZYJAOD/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
On Tue, Apr 21, 2020 at 2:09 AM Antoine Pitrou wrote: > > On Mon, 20 Apr 2020 15:30:37 -0700 > Nathaniel Smith wrote: > > > > tl;dr: accepting PEP 554 is effectively a C API break, and will force > > many thousands of people worldwide to spend many hours wrangling with > > subinterpreter support. > > For the record, that's not my reading of the PEP. PEP 554 doesn't > mandate that C extensions be rewritten to be subinterpreter-compliant. > It makes that an ideal, not an obligation. They would not have to be rewritten, unless they want to be used in subinterpreters. The PEP does not say this explicitly. Here are the options for handling non-compliant extension modules: 1. do nothing (extensions may break in ways that are hard for users to deal with) 2. raise ImportError if an extension without PEP 489 support is imported in a subinterpreter 3. add a per-interpreter isolated mode (default) that provides the behavior of #2, where non-isolated gives #1 At this point my plan was go with #2, to avoid giving extension authors the pain that comes with #1. If we went with #3, at least during the provisional phase, it would allow extension authors to try out their extensions in subinterpreters. -eric ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/E4456ZIXUK7O2QZ4DJETMKTSFYADFL7B/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
Hi Nathaniel, Thanks for your very interesting analysis :-) Le ven. 17 avr. 2020 à 23:12, Nathaniel Smith a écrit : > - The asyncio designers (esp. Guido) did a very extensive analysis of > these libraries' design choices, spoke to the maintainers about what > they'd learned from hard experience, etc. asyncio and Twisted could perfectly live outside CPython code base. asyncio even started as a 3rd party project named Tulip for Python 3.3! Subinterpreters are different: the implementation rely a lot on CPython internals. Developing it outside CPython would be way harder if not impossible. For example, Eric had to add PyThreadState.id and PyInterpreterState.id members. He made many changes in Py_NewInterpreter() and Py_EndInterpreter() which are part of Python internals. Moreover, there are many large refactoring changes which are required to properly implement subinterpreters, especially to better isolate each others: _PyRuntimeState structure to better identify shared globals, PyConfig API (PEP 587), move more structures per-interpreter (GC state, warnings state, pending calls, etc.). > - Even today, the limitations imposed by the stdlib release cycle > still add substantial difficulty to maintaining asyncio From what I understood, asyncio has a very large API since it has tons of features (subprocesses, transports and protocols, streams, futures and tasks, etc.) and requires a lot of glue code to provide the same API on all platforms which is really hard. I understood that the subinterpreters module is very light: just a few functions to provide "channels" and that's it (in short). $ ./python Python 3.9.0a5+ (heads/master:6a9e80a931, Apr 21 2020, 02:28:15) >>> import _xxsubinterpreters >>> len(dir(_xxsubinterpreters)) 30 >>> import asyncio >>> len(dir(asyncio)) 122 >>> dir(_xxsubinterpreters) ['ChannelClosedError', 'ChannelEmptyError', 'ChannelError', 'ChannelID', 'ChannelNotEmptyError', 'ChannelNotFoundError', 'InterpreterID', 'RunFailedError', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', '_channel_id', 'channel_close', 'channel_create', 'channel_destroy', 'channel_list_all', 'channel_recv', 'channel_release', 'channel_send', 'create', 'destroy', 'get_current', 'get_main', 'is_running', 'is_shareable', 'list_all', 'run_string'] > OTOH, AFAICT the new concurrency model in PEP 554 has never actually > been used, and it isn't even clear whether it's useful at all. Antoine (BDFL-delegate of the PEP) wants to mark the PEP as provisional: it means that we *can* still remove the whole module later if we decide that it's not worth it. The same was done with asyncio. By the way, Antoine was also the BDFL-delegate of the heavy PEP 3156 (asyncio) ;-) > Designing useful concurrency models is *stupidly* hard. And on top of > that, it requires major reworks of the interpreter internals + > disrupts the existing C extension module ecosystem -- which is very > different from asyncio, where folks who didn't use it could just > ignore it. A lot of work done to isolate subinterpreters also helps to "fix" CPython internals. For example, converting C extension modules to multiphase initialization (PEP 489) and add a module state helps to destroy all module objects at exit and also helps to break reference cycles in the garbage collector. It helps to fix this issue created in 2007: "Py_Finalize() doesn't clear all Python objects at exit" https://bugs.python.org/issue1635741 It also enhances the implementation of the "embed Python" use case: "leak" less objects at exit, better cleanup things, etc. That's why I'm supporter of the overall project: even if subinterpreters never take off, the implementation will make CPython better ;-) FYI it even helps indirectly my project to clean the C API ;-) PEP 554 implementation looks quite small: 2 800 lines of C code (Modules/_xxsubinterpretersmodule.c and Objects/interpreteridobject.c) and 2 100 lines of tests (Lib/test/test__xxsubinterpreters.py). Compare it to asyncio: 13 000 lines (Lib/asyncio/*.py) of Python and 20 800 lines of tests (Lib/test/test_asyncio/*.py). Victor -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/LUV4AHNVK4LX2FPUTVQUEU7GYZ5HPDVY/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
On 21/04/20 8:34 pm, Ronald Oussoren via Python-Dev wrote: As far as I understand proper support for subinterpreters also requires moving away from static type definition to avoid sharing objects > between interpreters (that is, use the PyType_FromSpec to build types). Which means updating every extension module that defines its own Python types and was written before PyType_FromSpec existed. I expect there is a huge number of those. -- Greg ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/BRDRFJFCCJAD6TIVYZMZGE7LDBDAIG3M/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
On Mon, 20 Apr 2020 19:21:21 -0600 Eric Snow wrote: > Honest question: how many C extensions have process-global state that > will cause problems under subinterpreters? In other words, how many > already break in mod_wsgi? A slightly tricky question is what happens if a PyObject survives longer than the subinterpreter that created it. For example, in PyArrow, we allow passing a Python buffer-like object as a C++ buffer to C++ APIs. The C++ buffer could conceivably be kept around by C++ code for some time. When the C++ buffer is destroyed, Py_DECREF() is called on the Python object (I figure that we would have to switch to the future interpreter-aware PyGILState API -- when will it be available?). But what happens if the interpreter which created the Python object is already destroyed at this point? Regards Antoine. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/4OW3D2ADPTJCEN5MALP3VKDBSOAACABI/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
> On 21 Apr 2020, at 03:21, Eric Snow wrote: > […] > On Mon, Apr 20, 2020 at 4:30 PM Nathaniel Smith wrote: > […] >> >> But notice that this means that no-one can use subinterpreters at all, >> until all of their C extensions have had significant reworks to use >> the new API, which will take years and tons of work -- it's similar to >> the Python 3 transition. Many libraries will never make the jump. > > Again, that is a grand statement that makes things sound much worse > than they really are. I expect very very few extensions will need > "significant reworks". Adding PEP 489 support will not take much > effort, on the order of minutes. Dealing with process-global state > will depend on how much, if any. > > Honest question: how many C extensions have process-global state that > will cause problems under subinterpreters? In other words, how many > already break in mod_wsgi? Fully supporting sub-interpreters in PyObjC will likely be a lot of work, mostly due to being able to subclass Objective-C classes from Python. With sub-interpreters a Python script in an interpreter could see an Objective-C class in a different sub-interpreter. The current PyObjC architecture assumes that there’s exactly one (sub-)interpreter, that’s probably fixable but is far from trivial. With the current API it might not even be possible to add sub-interpreter support (although I write this without having read the PEP). As far as I understand proper support for subinterpreters also requires moving away from static type definitions to avoid sharing objects between interpreters (that is, use the PyType_FromSpec to build types). At first glance this API does not support everything I do in PyObjC (fun with metaclasses, in C code). BTW. I don’t have a an opinion on the PEP itself at this time, mostly because it doesn’t match my current use cases. Ronald — Twitter / micro.blog: @ronaldoussoren Blog: https://blog.ronaldoussoren.net/ ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/IK3SSUVHZ3B6G77X6PNYDXMA42TFD23B/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
On Tue, 21 Apr 2020 19:45:02 +1200 Greg Ewing wrote: > On 21/04/20 11:24 am, Edwin Zimmerman wrote: > > Socket connections could be passed off from the main interpreter to > > sub-interpreters for concurrent processing that simply isn't possible > > with the global GIL > > I would need convincing that the processing involved things > that are truly held up by the GIL, rather than things like > blocking I/O calls that can release the GIL. In many realistic situations, it will be a mix of both. For example, parsing HTTP headers isn't trivial and will consume CPU time with the GIL taken, if written in pure Python. Regards Antoine. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/FZID7JKTABMVUWS4KV6JFPMNXSZPGQZO/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
On Mon, 20 Apr 2020 15:30:37 -0700 Nathaniel Smith wrote: > > tl;dr: accepting PEP 554 is effectively a C API break, and will force > many thousands of people worldwide to spend many hours wrangling with > subinterpreter support. For the record, that's not my reading of the PEP. PEP 554 doesn't mandate that C extensions be rewritten to be subinterpreter-compliant. It makes that an ideal, not an obligation. Regards Antoine. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/RFLPXB4ZD6SA5PS36ZJT6PTMOR6IGNUP/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
On 21/04/20 11:24 am, Edwin Zimmerman wrote: Socket connections could be passed off from the main interpreter to sub-interpreters for concurrent processing that simply isn't possible with the global GIL I would need convincing that the processing involved things that are truly held up by the GIL, rather than things like blocking I/O calls that can release the GIL. -- Greg ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/PLE7T4UXJI2XOI4C2GVT4PNFMTG7QFFY/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
Nathaniel, Your tone and approach to this conversation concern me. I appreciate that you have strong feelings here and readily recognize I have my own biases, but it's becoming increasingly hard to draw any constructive insight from what tend to be very longs posts from you. It ends up being a large commitment of time for small gains. And honestly, it's also becoming hard to not counter some of your more elaborate statements with my own unhelpful prose. In the interest of making things better, please take it all down a notch or two. I apologize if I sound frustrated. I am frustrated, which is only more frustrating because I respect you a lot and feel like your feedback should be more helpful. I'm trying to moderate my responses, but I expect some of my emotion may slip through. :/ On Mon, Apr 20, 2020 at 4:30 PM Nathaniel Smith wrote: > On Fri, Apr 17, 2020 at 3:57 PM Eric Snow wrote: > That makes it worse, right? If I wrote a PEP saying "here's some > features that could possibly someday be used to make a new concurrency > model", that wouldn't make it past the first review. Clearly, tying this to "concurrency models" is confusing here. So let's just say, as Paul Moore put it, the PEP allows us to "organize" our code in a new way (effectively along the lines of isolated threads with message passing). > I guess your perspective is, subinterpreters are already a CPython > feature, so we're not adding anything, and we don't really need to > talk about whether CPython should support subinterpreters. > > But this simply isn't true. Yes, there's some APIs for subinterpreters > added back in the 1.x days, but they were never really thought > through, and have never actually worked. The C-API was thought through more than sufficiently. Subinterpreters are conceptually and practically a very light wrapper around the fundamental architecture of CPython's runtime. The API exposes exactly that, no more, no less. What is missing or broken? They also work fine in most cases. Mostly they have problems with extension modules that have unsafe process-global state and break in some less common cases due to bugs in CPython (which have not been fixed because no one cared enough). > There are exactly 3 users, > and all have serious issues, and a strategy for avoiding > subinterpreters because of the brokenness. In practice, the existing > ecosystem of C extensions has never supported subinterpreters. Catch-22: why would they ever bother if no one is using them. > This is clearly not a great state of affairs – we should either > support them or not support them. Shipping a broken feature doesn't > help anyone. But the current status isn't terribly harmful, because > the general consensus across the ecosystem is that they don't work and > aren't used. > > If we start exposing them in the stdlib and encouraging people to use > them, though, that's a *huge* change. You are arguing that this is effectively a new feature. As you noted earlier, I am saying it isn't. > Our users trust us. If we tell > them that subinterpreters are a real thing now, then they'll spend > lots of effort on trying to support them. What is "lots"? We've yet to see clear evidence of possible severe impact. On the contrary, I've gotten feedback from folks highly involved in the ecosystem that it will not be a big problem. It won't take care of itself, but it won't require a massive effort. > Since subinterpreters are confusing, and break the C API/ABI How are they confusing and how do they break either the C-API or C-ABI? This sort of misinformation (or perhaps just miscommunication) is not helpful at all to your argument. >, this > means that every C extension author will have to spend a substantial > amount of time figuring out what subinterpreters are, how they work, > squinting at PEP 489, asking questions, auditing their code, etc. You make it sounds like tons of work, but I'm unconvinced, as noted earlier. Consider that we regularly have new features for which extensions must provide support. How is this different? > This > will take years, and in the mean time, users will expect > subinterpreters to work, be confused at why they break, yell at random > third-party maintainers, spend days trying to track down mysterious > problems that turn out to be caused by subinterpreters, etc. There > will be many many blog posts trying to explain subinterpreters and > understand when they're useful (if ever), arguments about whether to > support them. Twitter threads. Production experiments. If you consider > that we have thousands of existing C extensions and millions of users, > accepting PEP 554 means forcing people you don't know to collectively > spend many person-years on subinterpreters. Again you're painting a hopeless picture, but so far it's no more than a picture that contrasts with other less negative feedback I've gotten. So yours comes off as unhelpful here. > Random story time: NumPy deprecated some C APIs
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
On Mon, Apr 20, 2020 at 5:36 PM Edwin Zimmerman wrote: > > On 4/20/2020 7:33 PM, Nathaniel Smith wrote: > > On Mon, Apr 20, 2020 at 4:26 PM Edwin Zimmerman > > wrote: > >> On 4/20/2020 6:30 PM, Nathaniel Smith wrote: > >>> We already have robust support for threads for low-isolation and > >>> subprocesses for high-isolation. Can you name some use cases where > >>> neither of these are appropriate and you instead want an in-between > >>> isolation – like subprocesses, but more fragile and with odd edge > >>> cases where state leaks between them? > >> I don't know if this has been mentioned before or not, but I'll bring it > >> up now: massively concurrent networking code on Windows. Socket > >> connections could be passed off from the main interpreter to > >> sub-interpreters for concurrent processing that simply isn't possible with > >> the global GIL (provided the GIL actually becomes per-interpreter). On > >> *nix you can fork, this would give CPython on Windows similar capabilities. > > Both Windows and Unix have APIs for passing sockets between related or > > unrelated processes -- no fork needed. On Windows, it's exposed as the > > socket.share method: > > https://docs.python.org/3/library/socket.html#socket.socket.share > > > > The APIs for managing and communicating between processes are > > definitely not the most obvious or simplest to use, but they're very > > mature and powerful, and it's a lot easier to wrap them up in a > > high-level API than it is to effectively reimplement process > > separation from scratch inside CPython. > > > > -n > +1 on not being most obvious or simplest to use. Not only that, but to use > it you have to write Windows-specific code. PEP 554 would provide a uniform, > cross-platform capability that I would choose any day over a random pile of > os-specific hacks. I mean, sure, if you've decided to build one piece of hypothetical software well and another badly, then the good one will be better than the bad one, but that doesn't really say much, does it? In real life, I don't see how it's possible to get PEP 554's implementation to the point where it works reliably and robustly – i.e., I just don't think the promises the PEP makes can actually be fulfilled. And even if you did, it would still be several orders of magnitude easier to build a uniform, robust, cross-platform API on top of tools like socket.share than it would be to force changes on every C extension. PEP 554 is hugely expensive; you can afford a *lot* of careful systems engineering while still coming in way under that budget. -n -- Nathaniel J. Smith -- https://vorpus.org ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/L63XKYQDFVCOCNZC2VN27KFW2C3NTBKZ/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
On 4/20/2020 7:33 PM, Nathaniel Smith wrote: > On Mon, Apr 20, 2020 at 4:26 PM Edwin Zimmerman > wrote: >> On 4/20/2020 6:30 PM, Nathaniel Smith wrote: >>> We already have robust support for threads for low-isolation and >>> subprocesses for high-isolation. Can you name some use cases where >>> neither of these are appropriate and you instead want an in-between >>> isolation – like subprocesses, but more fragile and with odd edge >>> cases where state leaks between them? >> I don't know if this has been mentioned before or not, but I'll bring it up >> now: massively concurrent networking code on Windows. Socket connections >> could be passed off from the main interpreter to sub-interpreters for >> concurrent processing that simply isn't possible with the global GIL >> (provided the GIL actually becomes per-interpreter). On *nix you can fork, >> this would give CPython on Windows similar capabilities. > Both Windows and Unix have APIs for passing sockets between related or > unrelated processes -- no fork needed. On Windows, it's exposed as the > socket.share method: > https://docs.python.org/3/library/socket.html#socket.socket.share > > The APIs for managing and communicating between processes are > definitely not the most obvious or simplest to use, but they're very > mature and powerful, and it's a lot easier to wrap them up in a > high-level API than it is to effectively reimplement process > separation from scratch inside CPython. > > -n +1 on not being most obvious or simplest to use. Not only that, but to use it you have to write Windows-specific code. PEP 554 would provide a uniform, cross-platform capability that I would choose any day over a random pile of os-specific hacks. --Edwin ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/PVLDRLGQ24OV6VM5OR3OFF7HEQY245BP/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
On 4/20/2020 6:30 PM, Nathaniel Smith wrote: > We already have robust support for threads for low-isolation and > subprocesses for high-isolation. Can you name some use cases where > neither of these are appropriate and you instead want an in-between > isolation – like subprocesses, but more fragile and with odd edge > cases where state leaks between them? I don't know if this has been mentioned before or not, but I'll bring it up now: massively concurrent networking code on Windows. Socket connections could be passed off from the main interpreter to sub-interpreters for concurrent processing that simply isn't possible with the global GIL (provided the GIL actually becomes per-interpreter). On *nix you can fork, this would give CPython on Windows similar capabilities. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/MNC6BO7GC6RZA5TI7UTPHJXYDQLQY7DA/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
On Mon, Apr 20, 2020 at 4:26 PM Edwin Zimmerman wrote: > > On 4/20/2020 6:30 PM, Nathaniel Smith wrote: > > We already have robust support for threads for low-isolation and > > subprocesses for high-isolation. Can you name some use cases where > > neither of these are appropriate and you instead want an in-between > > isolation – like subprocesses, but more fragile and with odd edge > > cases where state leaks between them? > I don't know if this has been mentioned before or not, but I'll bring it up > now: massively concurrent networking code on Windows. Socket connections > could be passed off from the main interpreter to sub-interpreters for > concurrent processing that simply isn't possible with the global GIL > (provided the GIL actually becomes per-interpreter). On *nix you can fork, > this would give CPython on Windows similar capabilities. Both Windows and Unix have APIs for passing sockets between related or unrelated processes -- no fork needed. On Windows, it's exposed as the socket.share method: https://docs.python.org/3/library/socket.html#socket.socket.share The APIs for managing and communicating between processes are definitely not the most obvious or simplest to use, but they're very mature and powerful, and it's a lot easier to wrap them up in a high-level API than it is to effectively reimplement process separation from scratch inside CPython. -n -- Nathaniel J. Smith -- https://vorpus.org ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/KMS6JEGPB62STE4SE7YWGFALNFUE2LUX/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
On Fri, Apr 17, 2020 at 3:57 PM Eric Snow wrote: > > On Fri, Apr 17, 2020 at 2:59 PM Nathaniel Smith wrote: > > I think some perspective might be useful here :-). > > > > The last time we merged a new concurrency model in the stdlib, it was > > asyncio. > > > > [snip] > > > > OTOH, AFAICT the new concurrency model in PEP 554 has never actually > > been used, and it isn't even clear whether it's useful at all. > > Perhaps I didn't word things quite right. PEP 554 doesn't provide a > new concurrency model so much as it provides functionality that could > probably be used as the foundation for one. That makes it worse, right? If I wrote a PEP saying "here's some features that could possibly someday be used to make a new concurrency model", that wouldn't make it past the first review. > Ultimately the module > proposed in the PEP does the following: > > * exposes the existing subinterpreters functionality almost as-is So I think this is a place where we see things really differently. I guess your perspective is, subinterpreters are already a CPython feature, so we're not adding anything, and we don't really need to talk about whether CPython should support subinterpreters. But this simply isn't true. Yes, there's some APIs for subinterpreters added back in the 1.x days, but they were never really thought through, and have never actually worked. There are exactly 3 users, and all have serious issues, and a strategy for avoiding subinterpreters because of the brokenness. In practice, the existing ecosystem of C extensions has never supported subinterpreters. This is clearly not a great state of affairs – we should either support them or not support them. Shipping a broken feature doesn't help anyone. But the current status isn't terribly harmful, because the general consensus across the ecosystem is that they don't work and aren't used. If we start exposing them in the stdlib and encouraging people to use them, though, that's a *huge* change. Our users trust us. If we tell them that subinterpreters are a real thing now, then they'll spend lots of effort on trying to support them. Since subinterpreters are confusing, and break the C API/ABI, this means that every C extension author will have to spend a substantial amount of time figuring out what subinterpreters are, how they work, squinting at PEP 489, asking questions, auditing their code, etc. This will take years, and in the mean time, users will expect subinterpreters to work, be confused at why they break, yell at random third-party maintainers, spend days trying to track down mysterious problems that turn out to be caused by subinterpreters, etc. There will be many many blog posts trying to explain subinterpreters and understand when they're useful (if ever), arguments about whether to support them. Twitter threads. Production experiments. If you consider that we have thousands of existing C extensions and millions of users, accepting PEP 554 means forcing people you don't know to collectively spend many person-years on subinterpreters. Random story time: NumPy deprecated some C APIs some years ago, a little bit before I got involved. Unfortunately, it wasn't fully thought through; the new APIs were a bit nicer-looking, but didn't enable any new features, didn't provide any path to getting rid of the old APIs, and in fact it turned out that there were some critical use cases that still required the old API. So in practice, the deprecation was never going anywhere; the old APIs work just as well and are never going to get removed, so spending time migrating to the new APIs was, unfortunately, a completely pointless waste of time that provided zero value to anyone. Nonetheless, our users trusted us, so lots and lots of projects spend substantial effort on migrating to the new API: figuring out how it worked, making PRs, reviewing them, writing shims to work across the old and new API, having big discussions about how to make the new API work with Cython, debating what to do about the cases where the new APIs were inadequate, etc. None of this served any purpose: they just did it because they trusted us, and we misled them. It's pretty shameful, honestly. Everyone meant well, but in retrospect it was a terrible betrayal of our users' trust. Now, that only affected projects that were using the NumPy C API, and even then, only developers who were diligent and trying to follow the latest updates; there were no runtime warnings, nothing visible to end-users, etc. Your proposal has something like 100x-1000x more impact, because you want to make all C extensions in Python get updated or at least audited, and projects that aren't updated will produce mysterious crashes, incorrect output, or loud error messages that cause users to come after the developers and demand fixes. Now maybe that's worth it. I think on net the Py3 transition was worth it, and that was even more difficult. But Py3 had an incredible amount of scrutiny and rationale. Here
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
On Sat, 18 Apr 2020 10:08:27 -0600 Eric Snow wrote: > On Sat, Apr 18, 2020, 01:16 Antoine Pitrou wrote: > > > On Fri, 17 Apr 2020 13:59:46 -0600 > > Eric Snow wrote: > > > > For bytes or buffer objects, I understand that you propose to share > > > > the exact same PyObject objects between interpreters, at least in the > > > > first implementation. > > > > > > > > It may be better to have one proxy object in each interpreter which > > > > would control which interpreters can read and which interpreters can > > > > write into the object. It's a similar but more complex issue than > > > > singletons. > > > > > > It isn't the same Python object. It is the buffer that gets shared > > > (for objects that support the buffer protocol). > > > > When such a buffer dies, in which interpreter is the original object's > > destructor called? > > > > In the original interpreter (via a "pending call"). Great, thank you! Regards Antoine. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/M2PWQXFCYJOPZTJPL664NJOPHQPEITIX/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
On Sat, Apr 18, 2020, 01:16 Antoine Pitrou wrote: > On Fri, 17 Apr 2020 13:59:46 -0600 > Eric Snow wrote: > > > For bytes or buffer objects, I understand that you propose to share > > > the exact same PyObject objects between interpreters, at least in the > > > first implementation. > > > > > > It may be better to have one proxy object in each interpreter which > > > would control which interpreters can read and which interpreters can > > > write into the object. It's a similar but more complex issue than > > > singletons. > > > > It isn't the same Python object. It is the buffer that gets shared > > (for objects that support the buffer protocol). > > When such a buffer dies, in which interpreter is the original object's > destructor called? > In the original interpreter (via a "pending call"). -eric > ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/MGAMVLARCQXQ2Z2O5MN5W324KF47ATCR/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
It is not clear to me (as an end-user of Python) what is the value of exposing subinterpreters at the Python level except for multi core concurrency, but that requires one GIL per subinterpreter. Paul Moore mentions “alternative code structuring” as a benefit, but I don’t know what that means in concrete terms. If Paul or anyone else would make a case for the subinterpreters sharing a single GIL, that would be great. I understand many PEPs have to do with reducing technical debt in CPython internals, which is great. But PEP 554 is about exposing a feature to Python end users, so without use cases for them I believe this PEP should be put in the back burner until such cases are articulated more clearly for the wider Python community. Thanks, Luciano On Sat, Apr 18, 2020 at 07:09 Paul Moore wrote: > On Sat, 18 Apr 2020 at 00:03, Eric Snow > wrote: > > > > On Fri, Apr 17, 2020 at 2:59 PM Nathaniel Smith wrote: > > > > I know you want folks to consider PEP 554 on its own merits, ignoring > > > the GIL-splitting work, but let's be realistic: purely as a > > > concurrency framework, there's at least a dozen more > > > mature/featureful/compelling options in the stdlib and on PyPI, and as > > > an isolation mechanism, subinterpreters have been around for >20 years > > > and in that time they've found 3 users and no previous champions. > > > Obviously the GIL stuff is the only reason PEP 554 might be worth > > > accepting. > > > > Saying it's "obviously" the "only" reason is a bit much. :) PEP 554 > > exposes existing functionality that hasn't been all that popular > > (until recently for some reason ) mostly because it is old, was > > never publicized (until recently), and involved using the C-API. As > > soon as folks learn about it they want it, for various reasons > > including (relative) isolation and reduced resource usage in > > large-scale deployment scenarios. It becomes even more attractive if > > you say subinterpreters allow you to work around the GIL in a single > > process, but that isn't the only reason. > > As some context in contrast to Nathaniel's view, I remember back > around 15 years ago, when I first discovered the subinterpreter > support in the C API, I was tremendously interested in it, and very > frustrated that it wasn't exposed in Python. (To be fair, I was still > using Perl at the time, and it reminded me of Perl's implementation of > fork on Windows using subinterpreters, so maybe I was primed to see > use cases for it). I never did enough extension coding to use it in C, > but I remained interested in it for some time. > > This PEP is very exciting for me, precisely because it gives me a > chance to explore subinterpreters from Python, which I've been > interested in for all those years. I've not really thought of it in > terms of concurrency, so the GIL isn't key to me, but more as an > alternative code structuring option. I don't know if I'll use it much > in production code, but being able to experiment and assess the > options is something I'm looking forward to. So I for one am grateful > to you for doing this work, and I'm in favour of it being in 3.9, just > so I can get my hands on it at last. > > I do think that making it initially provisional may be a sensible > option, as it gives us the option to refine the Python API while > people are still learning how to use the new model. > > Paul > ___ > Python-Dev mailing list -- python-dev@python.org > To unsubscribe send an email to python-dev-le...@python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/LIGC4BUYDUU7JIH7MCJLNAEQYKFW2LHX/ > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Luciano Ramalho | Author of Fluent Python (O'Reilly, 2015) | http://shop.oreilly.com/product/0636920032519.do | Technical Principal at ThoughtWorks | Twitter: @ramalhoorg ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/UNCDNFDQBBPWYFY45T3VQ4VIXVE3SDGO/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
On Sat, 18 Apr 2020 at 00:03, Eric Snow wrote: > > On Fri, Apr 17, 2020 at 2:59 PM Nathaniel Smith wrote: > > I know you want folks to consider PEP 554 on its own merits, ignoring > > the GIL-splitting work, but let's be realistic: purely as a > > concurrency framework, there's at least a dozen more > > mature/featureful/compelling options in the stdlib and on PyPI, and as > > an isolation mechanism, subinterpreters have been around for >20 years > > and in that time they've found 3 users and no previous champions. > > Obviously the GIL stuff is the only reason PEP 554 might be worth > > accepting. > > Saying it's "obviously" the "only" reason is a bit much. :) PEP 554 > exposes existing functionality that hasn't been all that popular > (until recently for some reason ) mostly because it is old, was > never publicized (until recently), and involved using the C-API. As > soon as folks learn about it they want it, for various reasons > including (relative) isolation and reduced resource usage in > large-scale deployment scenarios. It becomes even more attractive if > you say subinterpreters allow you to work around the GIL in a single > process, but that isn't the only reason. As some context in contrast to Nathaniel's view, I remember back around 15 years ago, when I first discovered the subinterpreter support in the C API, I was tremendously interested in it, and very frustrated that it wasn't exposed in Python. (To be fair, I was still using Perl at the time, and it reminded me of Perl's implementation of fork on Windows using subinterpreters, so maybe I was primed to see use cases for it). I never did enough extension coding to use it in C, but I remained interested in it for some time. This PEP is very exciting for me, precisely because it gives me a chance to explore subinterpreters from Python, which I've been interested in for all those years. I've not really thought of it in terms of concurrency, so the GIL isn't key to me, but more as an alternative code structuring option. I don't know if I'll use it much in production code, but being able to experiment and assess the options is something I'm looking forward to. So I for one am grateful to you for doing this work, and I'm in favour of it being in 3.9, just so I can get my hands on it at last. I do think that making it initially provisional may be a sensible option, as it gives us the option to refine the Python API while people are still learning how to use the new model. Paul ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/LIGC4BUYDUU7JIH7MCJLNAEQYKFW2LHX/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
On Fri, 17 Apr 2020 13:59:46 -0600 Eric Snow wrote: > > For bytes or buffer objects, I understand that you propose to share > > the exact same PyObject objects between interpreters, at least in the > > first implementation. > > > > It may be better to have one proxy object in each interpreter which > > would control which interpreters can read and which interpreters can > > write into the object. It's a similar but more complex issue than > > singletons. > > It isn't the same Python object. It is the buffer that gets shared > (for objects that support the buffer protocol). When such a buffer dies, in which interpreter is the original object's destructor called? Regards Antoine. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/NAPY6WAIW2YWZROHX3BNOPXW6GGT42U3/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
On Fri, Apr 17, 2020 at 2:59 PM Nathaniel Smith wrote: > I think some perspective might be useful here :-). > > The last time we merged a new concurrency model in the stdlib, it was asyncio. > > [snip] > > OTOH, AFAICT the new concurrency model in PEP 554 has never actually > been used, and it isn't even clear whether it's useful at all. Perhaps I didn't word things quite right. PEP 554 doesn't provide a new concurrency model so much as it provides functionality that could probably be used as the foundation for one. Ultimately the module proposed in the PEP does the following: * exposes the existing subinterpreters functionality almost as-is * provides a minimal way to pass information between subinterpreters (which you don't need in C but do in Python code) * adds a few minor conveniences like propagating exceptions and making it easier to share buffers safely So the comparison with asyncio isn't all that fair. > Designing useful concurrency models is *stupidly* hard. And on top of > that, it requires major reworks of the interpreter internals + Nearly all of the "rework" is worth doing for other good reasons. Furthermore, I'd call at most a few of the reworks "major". > disrupts the existing C extension module ecosystem -- which is very > different from asyncio, where folks who didn't use it could just > ignore it. The concern is users opening issues saying "your extension won't work in subinterpreters", right? (You brought up the possible burden on extension authors in discussions several years ago, for which I'm still grateful.) How is that different from any other feature, new or not? "Your library doesn't provide an awaitable API." "Your library doesn't support pickling." "Your library doesn't implement the buffer protocol." PEP 554 doesn't introduce some new kind of impact. So (unless I've misunderstood), by your reasoning we wouldn't add any new features for which library authors might have to change something. Are you suggesting that the burden from PEP 554 will be larger than saying "then don't try to use our extension in subinterpreters"? Are you concerned about users reporting bugs that surface when an incompatible extension is used in a subinterpreter? That shouldn't be a problem if we raise ImportError if an extension that does not support PEP 489 is imported in a subinterpreter. FWIW, the impact to extension authors is the one thing about which I still have any meaningful uncertainty and worry. Various people have explained to me how it won't be a big problem, but I'm still nervous about it. I just don't think my worry is large than the actual risk (and possible cost). > So to me, it's kind of shocking that you'd even bring up the > possibility of merging PEP 554 as-is, without even a provisional > marker. Unsurprisingly it isn't shocking to me. :) From my point of view it seems okay. However, I'll be the first to recognize how hard it can be to see things from a different perspective. Hence I started this thread. :) > And if it's possible for it to live on PyPI, then why would we > even consider putting it into the stdlib? Given that we're exposing functionality of the CPython runtime I don't see the point in keeping this out of the stdlib. Furthermore, there are use cases to explore for subinterpreters in our test suite that we can address only if the "interpreters" module is part of the CPython repo. So why keep it hidden away and then publish the exact same thing on PyPI? > Personally, I'm still > leaning towards thinking that the whole subinterpreter project is > fundamentally flawed, and that on net we'd be better off removing > support for them entirely. By which I imagine you mean drop the subinterpreters API and not actually get rid of all the architecture related to PyInterpreterState (which is valuable from a code health perspective). > But that's a more complex and nuanced > question that I'm not 100% certain of, while the idea of merging it > for 3.9 seems like a glaringly obvious bad idea. Yeah, I remember your position from previous conversations (and still appreciate your feedback). :) > I know you want folks to consider PEP 554 on its own merits, ignoring > the GIL-splitting work, but let's be realistic: purely as a > concurrency framework, there's at least a dozen more > mature/featureful/compelling options in the stdlib and on PyPI, and as > an isolation mechanism, subinterpreters have been around for >20 years > and in that time they've found 3 users and no previous champions. > Obviously the GIL stuff is the only reason PEP 554 might be worth > accepting. Saying it's "obviously" the "only" reason is a bit much. :) PEP 554 exposes existing functionality that hasn't been all that popular (until recently for some reason ) mostly because it is old, was never publicized (until recently), and involved using the C-API. As soon as folks learn about it they want it, for various reasons including (relative) isolation and reduced
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
On Fri, Apr 17, 2020 at 11:50 AM Eric Snow wrote: > Dilemma > > > Many folks have conflated PEP 554 with having a per-interpreter GIL. > In fact, I was careful to avoid any mention of parallelism or the GIL > in the PEP. Nonetheless some are expecting that when PEP 554 lands we > will reach multi-core nirvana. > > While PEP 554 might be accepted and the implementation ready in time > for 3.9, the separate effort toward a per-interpreter GIL is unlikely > to be sufficiently done in time. That will likely happen in the next > couple months (for 3.10). > > So...would it be sufficiently problematic for users if we land PEP 554 > in 3.9 without per-interpreter GIL? > > Options > > > Here are the options as I see them (if the PEP is accepted in time for 3.9): > > 1. merge PEP 554 into 3.9 even if per-interpreter GIL doesn't get into > 3.9 (they get parallelism for free in 3.10) > 2. like 1, but mark the module as provisional until per-interpreter GIL lands > 3. do not merge PEP 554 until per-interpreter GIL is merged > 4. like 3, but publish a 3.9-only module to PyPI in the meantime I think some perspective might be useful here :-). The last time we merged a new concurrency model in the stdlib, it was asyncio. In that case, the process went something like: - We started with two extremely mature libraries (Twisted + Tornado) with long histories of real-world use - The asyncio designers (esp. Guido) did a very extensive analysis of these libraries' design choices, spoke to the maintainers about what they'd learned from hard experience, etc. - Asyncio was initially shipped outside the stdlib to allow for testing and experimentation, and at this stage it was used to build non-trivial projects (e.g. the aiohttp project's first commits use tulip, not asyncio) - When it was eventually added to the stdlib, it was still marked provisional for multiple python releases, and underwent substantial and disruptive changes during this time - Even today, the limitations imposed by the stdlib release cycle still add substantial difficulty to maintaining asyncio OTOH, AFAICT the new concurrency model in PEP 554 has never actually been used, and it isn't even clear whether it's useful at all. Designing useful concurrency models is *stupidly* hard. And on top of that, it requires major reworks of the interpreter internals + disrupts the existing C extension module ecosystem -- which is very different from asyncio, where folks who didn't use it could just ignore it. So to me, it's kind of shocking that you'd even bring up the possibility of merging PEP 554 as-is, without even a provisional marker. And if it's possible for it to live on PyPI, then why would we even consider putting it into the stdlib? Personally, I'm still leaning towards thinking that the whole subinterpreter project is fundamentally flawed, and that on net we'd be better off removing support for them entirely. But that's a more complex and nuanced question that I'm not 100% certain of, while the idea of merging it for 3.9 seems like a glaringly obvious bad idea. I know you want folks to consider PEP 554 on its own merits, ignoring the GIL-splitting work, but let's be realistic: purely as a concurrency framework, there's at least a dozen more mature/featureful/compelling options in the stdlib and on PyPI, and as an isolation mechanism, subinterpreters have been around for >20 years and in that time they've found 3 users and no previous champions. Obviously the GIL stuff is the only reason PEP 554 might be worth accepting. Or if PEP 554 is really a good idea on its own merits, purely as a new concurrency API, then why not build that concurrency API on top of multiprocessing and put it on PyPI and let real users try it out? One more thought. Quoting from Poul Henning-Kemp's famous email at bikeshed.org: > Parkinson shows how you can go in to the board of directors and get approval for building a multi-million or even billion dollar atomic power plant, but if you want to build a bike shed you will be tangled up in endless discussions. > > Parkinson explains that this is because an atomic plant is so vast, so expensive and so complicated that people cannot grasp it, and rather than try, they fall back on the assumption that somebody else checked all the details before it got this far. Richard P. Feynmann gives a couple of interesting, and very much to the point, examples relating to Los Alamos in his books. > > A bike shed on the other hand. Anyone can build one of those over a weekend, and still have time to watch the game on TV. So no matter how well prepared, no matter how reasonable you are with your proposal, somebody will seize the chance to show that he is doing his job, that he is paying attention, that he is *here*. Normally, when people reference this story they focus on the bikeshed, hence the term "bikeshedding". But for PEP 554, you're building a nuclear power plant :-). The whole conglomeration of a new
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
On Fri, Apr 17, 2020 at 1:54 PM Victor Stinner wrote: > Honestly, I don't think that we are far from being able to move the > GIL lock per interpreter. But this change alone is useless, since we > still have too many C extensions which are not ready for that: Agreed. This is a big part of why I don't think we can make the GIL per-interpreter for 3.9. However, there are some things we could do to mitigate the problem if we want to aim for 3.9: * only allow the main interpreter to import extension modules that do not implement PEP 489 support + in subinterpreters it would be an ImportError * allow subinterpreters to be created either isolated (not sharing the GIL, etc.) or not isolated (sharing the GIL, etc.) + the default is "not isolated" (but only while the "interpreters" module is provisional) and users must opt in to isolated mode + this would allow extension authors to more easily check their modules for subinterpreter support, without causing excessive bug reports from users trying out subinterpreters I also am not convinced that in the next month we can work out all the possible corner cases problems that I expect will pop up once we have a per-interpreter GIL. Then again, arguably that could be addressed after the feature freeze if we think those possible issues could be sorted out in a month or two. :) > I would prefer PEP 554 to be restricted to the communication between > interpreters. > > Isolating subinterpreters should be out of the scope of this PEP, > since it's a different problem and we cannot solve all problems in a > single PEP. Isolating interpreters isn't part of PEP 554, other than that objects are not shared between interpreters. > Not sure if a PEP is needed to isolate subinterpreters. There are > already many issues and > https://github.com/ericsnowcurrently/multi-core-python/ to track the > work. Agreed. I don't know that anyone had or has any intention of adding a PEP related to isolation (i.e. moving the GIL to PyInterpreterState). -eric ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/QBY4JIXMYNWJOY3POH6YV35FWI2H6Z7I/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
On Fri, Apr 17, 2020 at 1:39 PM Victor Stinner wrote: > Sharing directly singletons like None can become a performance kill > once subinterpreters will run in parallel: > https://bugs.python.org/issue39511 > > Mark Shannon summarized: "Having two CPUs write to the same cache line > is a well known performance problem. There's nothing special about > CPython here. The proper name for it seems to be "cache line > ping-pong", but a search for "false sharing" might be more > informative." > > The problem is that PyObject.ob_refcnt should be protected by atomic > operations or locks if these objects are shared directly. I agree. Sharing objects between subinterpreters is a recipe for pain. That's why the PEP says we will not. :) > I proposed to have singletons per interpreter, some people would > prefer immortal singletons. Yeah, we definitely have to sort that out before we make the GIL per-interpreter. > For bytes or buffer objects, I understand that you propose to share > the exact same PyObject objects between interpreters, at least in the > first implementation. > > It may be better to have one proxy object in each interpreter which > would control which interpreters can read and which interpreters can > write into the object. It's a similar but more complex issue than > singletons. It isn't the same Python object. It is the buffer that gets shared (for objects that support the buffer protocol). However, I suppose that it will have a similar problem to what you described above, since it is the same (read-only) memory in both interpreters. I'm not too worried about that at the moment though, particularly if we make the module provisional. -eric ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/FT6UOCKNOHAZVNNU6V5X6TYWCXFPWMWM/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
Le ven. 17 avr. 2020 à 20:56, Eric Snow a écrit : > keep in mind that > subinterpreters currently have various other limitations (aside from > sharing the GIL): > > * nothing about them has been optimized (e.g. interpreter startup, > data sharing, data passing) > * extension modules with process-global state may break > * some bugs still remain in the corners Honestly, I don't think that we are far from being able to move the GIL lock per interpreter. But this change alone is useless, since we still have too many C extensions which are not ready for that: * C extension modules should be converted to multiphase initialization (PEP 489) * Statically allocated types should be converted to heap allocated types (PyType_FromSpec) * A module state should be added to a bunch of C extension modules * Many C extension modules don't implement the GC protocol, since it wasn't really needed previously: this should now be done (implement traverse, clear, free) * These two changes require PEP 573 to be implemented to get access the module state in functions where it wasn't possible previously The good news is that there is an on-going effort to complete this task: https://bugs.python.org/issue1635741 More and more C extension modukes are converted. The bad news is that better isolating subinterpreters "suddently" triggers old issues. See for example https://bugs.python.org/issue40217 "The garbage collector doesn't take in account that objects of heap allocated types hold a strong reference to their type" which is a very complex issue. I also fixed many reference cycles and other bugs, discovered after some specific C extensions were modified. There are still tons of "static" variables all around the code and still no technical solution to have "per interpreter" variable. We need something like Thread-Locale State, but for interpreters in the C API. PyInterpreter.dict is not convenient in C. Well, I'm not writing that for Eric who is already well aware of these issues, but for others ;-) > At this point I think we should go with option #1 (or possibly #2). > IMHO, we would be fine merging PEP 554 without per-interpreter GIL. I would prefer PEP 554 to be restricted to the communication between interpreters. Isolating subinterpreters should be out of the scope of this PEP, since it's a different problem and we cannot solve all problems in a single PEP. See what happened with PEP 432 which tried to solve multiple related problems: PEP 587 fixed the preinitialization and the configuration, but not the "path configuration" issue. Excluding the "path configuration" from the PEP 432 in the PEP 587 made possible to get a PEP "done". Not sure if a PEP is needed to isolate subinterpreters. There are already many issues and https://github.com/ericsnowcurrently/multi-core-python/ to track the work. Victor -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/U65UDU5V64EPYHPBLQKOEU3A23ZXIBY7/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
On Fri, Apr 17, 2020 at 1:39 PM Victor Stinner wrote: > I have concerns about shared data. > > The current implementation of CPython and subinterpreters still shares > singletons. Moreover, I understand that the PEP proposes to first > directly share PyObject between subinterpreters, like bytes objects. Hmm, the PEP explicitly says: Note that objects are not shared between interpreters since they are tied to the interpreter in which they were created. Instead, the objects' data is passed between interpreters. See the Shared data section for more details about sharing between interpreters. > Isolating subinterpreters is a big task and so all issues cannot be > solved at once. So I am not with making some compromises for a first > milestone. +1 To me, one of the most important things is to get the functionality out there as soon as possible, even if it is limited (but not useless). > The problem is to define which semantics can be exposed in the public > API for the first milestone. > > [snip] > > What is the plan for Python 3.9? What semantics would be exposed in Python > 3.9? > > Should we wait until these questions are answered before making the API > public? The PEP proposes minimal functionality. We can relax that in future versions of Python on a case-by-case basis. If you are specifically referring to object sharing, the PEP already says there is none. -eric ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/QVCKZVUECJQ3DCXO2GQ7HZA4HNAWB734/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
On Fri, Apr 17, 2020 at 1:38 PM Brett Cannon wrote: > Eric Snow wrote: > > 1. merge PEP 554 into 3.9 even if per-interpreter GIL doesn't get into 3.9 > > (they get parallelism for free in 3.10) > > 2. like 1, but mark the module as provisional until per-interpreter GIL > > lands > > 3. do not merge PEP 554 until per-interpreter GIL is merged > > 4. like 3, but publish a 3.9-only module to PyPI in the meantime > > My vote is for #2. No need to rush it and it gives you feedback on the API > from the public before it gets locked in for backwards-compatibility. Yeah, I'd be fine with that. Actually, making the module provisional is already part of the PEP. :) > And you do know you could have done this as a poll on Discourse, right? Yeah, I thought of that after the fact. Then again, discourse isn't the tool I already had open. :) -eric ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/CDVYBRIZ77BJN7WGGSLWULZ22SLB2TB5/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
Hi Eric, Le ven. 17 avr. 2020 à 20:56, Eric Snow a écrit : > With the 3.9 feature freeze coming up I'm considering options for PEP > 554. I'm hopeful to have a pronouncement from Antoine in the near > future. If that comes in time for 3.9, we will have the > implementation ready to go. It is already nearly complete and the > remaining work should be done in the next week or so. I have concerns about shared data. The current implementation of CPython and subinterpreters still shares singletons. Moreover, I understand that the PEP proposes to first directly share PyObject between subinterpreters, like bytes objects. Isolating subinterpreters is a big task and so all issues cannot be solved at once. So I am not with making some compromises for a first milestone. The problem is to define which semantics can be exposed in the public API for the first milestone. -- Sharing directly singletons like None can become a performance kill once subinterpreters will run in parallel: https://bugs.python.org/issue39511 Mark Shannon summarized: "Having two CPUs write to the same cache line is a well known performance problem. There's nothing special about CPython here. The proper name for it seems to be "cache line ping-pong", but a search for "false sharing" might be more informative." The problem is that PyObject.ob_refcnt should be protected by atomic operations or locks if these objects are shared directly. I proposed to have singletons per interpreter, some people would prefer immortal singletons. -- For bytes or buffer objects, I understand that you propose to share the exact same PyObject objects between interpreters, at least in the first implementation. It may be better to have one proxy object in each interpreter which would control which interpreters can read and which interpreters can write into the object. It's a similar but more complex issue than singletons. -- What is the plan for Python 3.9? What semantics would be exposed in Python 3.9? Should we wait until these questions are answered before making the API public? Victor > I'm looking for feedback on a nagging worry I have with merging PEP > 554 into 3.9 (if it is accepted) without having the per-interpreter > GIL. See below. I'll update the PEP later with relevant info from > this discussion. > > wishing-you-were-all-here-ly yours, > > -eric > > + (++ > > Dilemma > > > Many folks have conflated PEP 554 with having a per-interpreter GIL. > In fact, I was careful to avoid any mention of parallelism or the GIL > in the PEP. Nonetheless some are expecting that when PEP 554 lands we > will reach multi-core nirvana. > > While PEP 554 might be accepted and the implementation ready in time > for 3.9, the separate effort toward a per-interpreter GIL is unlikely > to be sufficiently done in time. That will likely happen in the next > couple months (for 3.10). > > So...would it be sufficiently problematic for users if we land PEP 554 > in 3.9 without per-interpreter GIL? > > Options > > > Here are the options as I see them (if the PEP is accepted in time for 3.9): > > 1. merge PEP 554 into 3.9 even if per-interpreter GIL doesn't get into > 3.9 (they get parallelism for free in 3.10) > 2. like 1, but mark the module as provisional until per-interpreter GIL lands > 3. do not merge PEP 554 until per-interpreter GIL is merged > 4. like 3, but publish a 3.9-only module to PyPI in the meantime > > Context > > > Like I said, PEP 554 purposefully does not talk about parallelism or > the GIL. Instead it focuses strictly on the following (with an > emphasis on minimalism): > > * exposing the existing C-API > * supporting a new concurrency model (CSP) > > Regarding that first one, the value is in exposing the existing > functionality to a broader group of people. keep in mind that > subinterpreters currently have various other limitations (aside from > sharing the GIL): > > * nothing about them has been optimized (e.g. interpreter startup, > data sharing, data passing) > * extension modules with process-global state may break > * some bugs still remain in the corners > > Currently, there is a chicken-and-egg problem. It is unlikely that > there will be sufficient motivation to address those limitations until > the community starts really using subinterpreters. Hence PEP 554. > > My Position > > > At this point I think we should go with option #1 (or possibly #2). > IMHO, we would be fine merging PEP 554 without per-interpreter GIL. > Here's why: > > * the two objectives of the PEP have value for the community (as > explained in the PEP) > * the sooner we can get the functionality of subinterpreters out to > the broader world, the better > * I don't think the lack of parallelism in 3.9 will trip anyone up > > My only concern is that folks try them out in 3.9, get frustrated by > the limitations, and then there is a mental barrier to trying
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
Eric Snow wrote: > Hi all, > It was nice to speak with many of you at the online language summit > this week. It was the next best thing to seeing you all in person! :) > With the 3.9 feature freeze coming up I'm considering options for PEP > > I'm hopeful to have a pronouncement from Antoine in the near > future. If that comes in time for 3.9, we will have the > implementation ready to go. It is already nearly complete and the > remaining work should be done in the next week or so. > > I'm looking for feedback on a nagging worry I have with merging PEP > 554 into 3.9 (if it is accepted) without having the per-interpreter > GIL. See below. I'll update the PEP later with relevant info from > this discussion. > wishing-you-were-all-here-ly yours, > -eric > + (++ > Dilemma > Many folks have conflated PEP 554 with having a per-interpreter GIL. > In fact, I was careful to avoid any mention of parallelism or the GIL > in the PEP. Nonetheless some are expecting that when PEP 554 lands we > will reach multi-core nirvana. > While PEP 554 might be accepted and the implementation ready in time > for 3.9, the separate effort toward a per-interpreter GIL is unlikely > to be sufficiently done in time. That will likely happen in the next > couple months (for 3.10). > So...would it be sufficiently problematic for users if we land PEP 554 > in 3.9 without per-interpreter GIL? > Options > Here are the options as I see them (if the PEP is accepted in time for 3.9): > > 1. merge PEP 554 into 3.9 even if per-interpreter GIL doesn't get into 3.9 > (they get parallelism for free in 3.10) > 2. like 1, but mark the module as provisional until per-interpreter GIL lands > 3. do not merge PEP 554 until per-interpreter GIL is merged > 4. like 3, but publish a 3.9-only module to PyPI in the meantime My vote is for #2. No need to rush it and it gives you feedback on the API from the public before it gets locked in for backwards-compatibility. And you do know you could have done this as a poll on Discourse, right? -Brett > > Context > Like I said, PEP 554 purposefully does not talk about parallelism or > the GIL. Instead it focuses strictly on the following (with an > emphasis on minimalism): > > exposing the existing C-API > supporting a new concurrency model (CSP) > > Regarding that first one, the value is in exposing the existing > functionality to a broader group of people. keep in mind that > subinterpreters currently have various other limitations (aside from > sharing the GIL): > > nothing about them has been optimized (e.g. interpreter startup, > data sharing, data passing) > extension modules with process-global state may break > some bugs still remain in the corners > > Currently, there is a chicken-and-egg problem. It is unlikely that > there will be sufficient motivation to address those limitations until > the community starts really using subinterpreters. Hence PEP 554. > My Position > At this point I think we should go with option #1 (or possibly #2). > IMHO, we would be fine merging PEP 554 without per-interpreter GIL. > Here's why: > > the two objectives of the PEP have value for the community (as > explained in the PEP) > the sooner we can get the functionality of subinterpreters out to > the broader world, the better > I don't think the lack of parallelism in 3.9 will trip anyone up > > My only concern is that folks try them out in 3.9, get frustrated by > the limitations, and then there is a mental barrier to trying them > again in the future when the limitations have been reduced or > eliminated. However, I don't think that will be a problem for people > that would benefit from serious use of subinterpreters. > Furthermore, if we're clear that "for now" subinterpreters still share > the GIL in 3.9 then I'm not so worried about folks getting confused. > In fact, as long as we're clear about all the current limitations then > there isn't much downside to option #1. > Marking the module as "provisional" may help communicate that there > are limitations that impact use. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/WCMLX56RRZYGTWQFIL2D2D3FHEI2CZI2/ Code of Conduct: http://python.org/psf/codeofconduct/
