[Python-Dev] Re: PEP 554 comments
Eric Snow wrote: > We will mark it "provisional" in the docs, which I expect will include > info on what that means and why it is provisional. If you'd like an example format for marking a section of the docs as provisional w/ reST, something like this at the top should suffice (with perhaps something more specific to the subinterpreters module): .. note:: This section of the documentation and all of its members have been added *provisionally*. For more details, see :term:`provisional api`. :term:`provisional api` generates a link to https://docs.python.org/3/glossary.html#term-provisional-api. On Tue, Apr 21, 2020 at 12:09 PM Eric Snow wrote: > > On Tue, Apr 21, 2020 at 7:25 AM Victor Stinner wrote: > > Would it make sense to start by adding the module as a private > > "_subinterpreters" module but document it? The "_" prefix would be a > > reminder that "hey! it's experimental, there is a no backward > > compatibility warranty there". > > I would expect a leading underscore to be confusing (as well as > conflicting with the name of the low-level module). if we did > anything then it would probably make more sense to name the module > something like "interpreters_experimental". However, I'm not sure > that offers that much benefit. > > > We can also add a big warning in the documentation. > > We will mark it "provisional" in the docs, which I expect will include > info on what that means and why it is provisional. > > -eric > ___ > Python-Dev mailing list -- python-dev@python.org > To unsubscribe send an email to python-dev-le...@python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/LB7ACPFAT3QU5YUPIFWH4NHB7YYVJXQG/ > Code of Conduct: http://python.org/psf/codeofconduct/ ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/TSIV524HRFGTKBPPV2IMO5J2U4QGDDKQ/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 554 for 3.9 or 3.10?
> On 21 Apr 2020, at 16:58, Eric Snow wrote: > > Thanks for explaining that, Ronald. It sounds like a lot of the > effort would relate to making classes work. I have some comments > in-line below. > > -eric > > On Tue, Apr 21, 2020 at 2:34 AM Ronald Oussoren > wrote: >>> On 21 Apr 2020, at 03:21, Eric Snow wrote: >>> Honest question: how many C extensions have process-global state that >>> will cause problems under subinterpreters? In other words, how many >>> already break in mod_wsgi? >> >> Fully supporting sub-interpreters in PyObjC will likely be a lot of work, >> mostly >> due to being able to subclass Objective-C classes from Python. With sub- >> interpreters a Python script in an interpreter could see an Objective-C >> class in >> a different sub-interpreter. The current PyObjC architecture assumes that >> there’s exactly one (sub-)interpreter, that’s probably fixable but is far >> from trivial. > > Are the Objective-C classes immutable? Are the wrappers stateful at > all? > Without context I'm not clear on how you would be impacted by > operation under subinterpreters (i.e. PEP 554), but it sounds like the > classes do have global state that is in fact interpreter-specific. I > expect you would also be impacted by subinterpreters not sharing the > GIL but that is a separate matter (see below). My mail left out some important information, sorry about that. PyObjC is a two-way bridge between Python and Objective-C. One half of this is that is bridging Objective-C classes (and instances) to Python. This is fairly straightforward, although the proxy objects are not static and can have methods defined in Python (helper methods that make the Objective-C classes nicer to use from Python, for example to define methods that make it possible to use an NSDictionary as if it were a regular Python dict). The other half is that it is possible to implement Objective-C classes in Python: class MyClass (Cocoa.NSObject): def anAction_(self, sender): … This defines a Python classes named “MyClass”, but also an Objective-C class of the same name that forwards Objective-C calls to Python. The implementation for this uses PyGILState_Ensure, which AFAIK is not yet useable with sub-interpreters. PyObjC also has Objective-C proxy classes for generic Python objects, making it possible to pass a normal Python dictionary to an Objective-C API that expects an NSDictionary instance. Things get interesting when combining the two with sub-interpreters: With the current implementation the Objective-C world would be a channel for passing “live” Python objects between sub-interpreters. The translation tables for looking up existing proxies (mapping from Python to Objective-C and vice versa) are currently singletons. This is probably fixable with another level of administration, by keeping track of the sub-interpreter that owns a Python object I could ensure that Python objects owned by a different sub-interpreter are proxied like any other Objective-C object which would close this loophole. That would require significant changes to a code base that’s already fairly complex, but should be fairly straightforward. > > Regardless, I expect there are others in a similar situation. It > would be good to understand your use case and help with a solution. > Is there a specific example you can point to of code that would be > problematic under subinterpreters? > >> With the current API it might not even be possible to add sub-interpreter >> support > > What additional API would be needed? See above, the main problem is PyGILState_Ensure. I haven’t spent a lot of time thinking about this though, I might find other issues when I try to support sub-interpreters. > >> (although I write this without having read the PEP). > > Currently PEP 554 does not talk about how to make extension modules > compatible with subinterpreters. That may be worth doing, though it > would definitely have to happen in the docs and (to an extent) the 3.9 > "What's New" page. There is already some discussion on what should be > in those docs (see > https://github.com/ericsnowcurrently/multi-core-python/issues/53). > > Note that, until the GIL becomes per-interpreter, sharing objects > isn't a problem. We were not planning on having a PEP for the > stop-sharing-the-GIL effort, but I'm starting to think that it may be > worth it, to cover the impact on extension modules (e.g. mitigations). > > So if you leave out the complications due to not sharing the GIL, the > main problem extension authors face with subinterpreters (exposed by > PEP 554) is when their module has process-global state that breaks > under subinterpreters. From your description above, it sounds like > you may be in that situation. > >> As far as I understand proper support for subinterpreters also requires >> moving >> away from static type definitions to avoid sharing objects between >> interpreters >> (that is, use t
[Python-Dev] Re: Comments on PEP 554 (Multiple Interpreters in the Stdlib)
FYI, I'm not ignoring you. :) Life intervened. I'll respond in the next day or two. -eric On Tue, Apr 21, 2020 at 10:42 AM Mark Shannon wrote: > > Hi, > > I'm generally in favour of PEP 554, but I don't think it is ready to be > accepted in its current form. > > My main objection is that without per-subinterpeter GILs (SILs?) PEP 554 > provides no value over threading or multi-processing. > Multi-processing provides true parallelism and threads provide shared > memory concurrency. > > If per-subinterpeter GILs are possible then, and only then, > sub-interpreters will provide true parallelism and (limited) shared > memory concurrency. > > The problem is that we don't know whether we can implement > per-subinterpeter GILs without too large a negative performance impact. > I think we can, but we can't say so for certain. > > So, IMO, we should not accept PEP 554 until we know for sure that > per-subinterpeter GILs can be implemented efficiently. > > > > Detailed critique > - > > I don't see how `list_all()` can be both safe and accurate. The Java > equivalent makes no guarantees of accuracy. > Attempting to lock the list is likely to lead to deadlock and not > locking it will lead to races; potentially dangerous ones. > I think it would be best to drop this. > > `list_all_channels()`. See `list_all()` above. > > `.destroy()` is either misleading or unsafe. > What does this do? > > >>> is.destroy() > >>> is.run() > > If `run()` raises an exception then the interpreter must exist. Rename > to `close()` perhaps? > > `Channel.interpreters` see `list_all()` and `list_all_channels()` above. > > How does `is_shareable()` work? Are you proposing some mechanism to > transfer an object from one sub-interpreter to another? How would that > work? If objects are not shared but serialized, why not use marshal or > pickle instead of inventing a third serialization protocol? > > It would be clearer if channels only dealt with simple, contiguous > binary data. As it stands the PEP doesn't state what form the received > object will take. > Once channels supporting the transfer of bytes objects work, then it is > simple to pass more complex objects using pickle or marshal. > > Channels need a more detailed description of their lifespan. Ideally a > state machine. > For example: > How does an interpreter detach from the receiving end of a channel that > is never empty? > What happens if an interpreter deletes the last reference to a non-empty > channel? On the receiving end, or on the sending end? > > Isn't the lack of buffering in channels a recipe for deadlocks? > > What is the mechanism for reliably copying exceptions from one > sub-interpreter to another in the `run()` method? If `run()` can raise > an exception, why not let it return values? > > > Cheers, > Mark. > ___ > Python-Dev mailing list -- python-dev@python.org > To unsubscribe send an email to python-dev-le...@python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/ZSE2G37E24YYLNMQKOQSBM46F7KLAOZF/ > Code of Conduct: http://python.org/psf/codeofconduct/ ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/CEZXKXQTKWM7RX3CVOAFUZTHRYSERNCZ/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Comments on PEP 554 (Multiple Interpreters in the Stdlib)
On 4/21/20 12:32 PM, Mark Shannon wrote: Hi, I'm generally in favour of PEP 554, but I don't think it is ready to be accepted in its current form. BTW, thanks for including the name of the PEP in the subject. As a casual reader of this list, it's very helpful to have more than just the number, so I can decide whether to read into the deeper details. --Ned. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/5J27XI3ILH3YARS6DZKXZDDS2DRF3BRY/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Comments on PEP 554 (Multiple Interpreters in the Stdlib)
On 22/04/2020 19:40, Ned Batchelder wrote: On 4/21/20 12:32 PM, Mark Shannon wrote: Hi, I'm generally in favour of PEP 554, but I don't think it is ready to be accepted in its current form. BTW, thanks for including the name of the PEP in the subject. As a casual reader of this list, it's very helpful to have more than just the number, so I can decide whether to read into the deeper details. Hear, hear! This is a point which is always worth bearing in mind. Whenever I send an email to my boss (usually containing a software update) I try to include all relevant "keywords" in the subject so that he can easily search for it months later. In other words I try to make my emails a valuable resource. Rob Cliffe ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/Q3N63NCWL4CBGYQ3GYHZW5SRYGJBSMLS/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Comments on PEP 554 (Multiple Interpreters in the Stdlib)
Mark Shannon wrote: > If `run()` can raise > an exception, why not let it return values? If there's not an implementation detail that makes this impractical, I'd like to give my +1 on the `Interpreter.run()` method returning values. From a usability perspective, it seems incredibly convenient to have the ability to call a function in a subinterpreter, and then directly get the return value instead of having to send the result through a channel (for more simple use cases). Also, not that the API for subinterpreters needs to be at all similar to asyncio, but it would be consistent with `asyncio.run()` with regards to being able to return values. Although one could certainly argue that `asyncio.run()` and `Interpreter.run()` will have significantly different use cases; with `asyncio.run()` being intended as a primary entry point for a program, and `Interpreter.run()` being used to execute arbitrary code in a single interpreter. On Tue, Apr 21, 2020 at 12:45 PM Mark Shannon wrote: > > Hi, > > I'm generally in favour of PEP 554, but I don't think it is ready to be > accepted in its current form. > > My main objection is that without per-subinterpeter GILs (SILs?) PEP 554 > provides no value over threading or multi-processing. > Multi-processing provides true parallelism and threads provide shared > memory concurrency. > > If per-subinterpeter GILs are possible then, and only then, > sub-interpreters will provide true parallelism and (limited) shared > memory concurrency. > > The problem is that we don't know whether we can implement > per-subinterpeter GILs without too large a negative performance impact. > I think we can, but we can't say so for certain. > > So, IMO, we should not accept PEP 554 until we know for sure that > per-subinterpeter GILs can be implemented efficiently. > > > > Detailed critique > - > > I don't see how `list_all()` can be both safe and accurate. The Java > equivalent makes no guarantees of accuracy. > Attempting to lock the list is likely to lead to deadlock and not > locking it will lead to races; potentially dangerous ones. > I think it would be best to drop this. > > `list_all_channels()`. See `list_all()` above. > > `.destroy()` is either misleading or unsafe. > What does this do? > > >>> is.destroy() > >>> is.run() > > If `run()` raises an exception then the interpreter must exist. Rename > to `close()` perhaps? > > `Channel.interpreters` see `list_all()` and `list_all_channels()` above. > > How does `is_shareable()` work? Are you proposing some mechanism to > transfer an object from one sub-interpreter to another? How would that > work? If objects are not shared but serialized, why not use marshal or > pickle instead of inventing a third serialization protocol? > > It would be clearer if channels only dealt with simple, contiguous > binary data. As it stands the PEP doesn't state what form the received > object will take. > Once channels supporting the transfer of bytes objects work, then it is > simple to pass more complex objects using pickle or marshal. > > Channels need a more detailed description of their lifespan. Ideally a > state machine. > For example: > How does an interpreter detach from the receiving end of a channel that > is never empty? > What happens if an interpreter deletes the last reference to a non-empty > channel? On the receiving end, or on the sending end? > > Isn't the lack of buffering in channels a recipe for deadlocks? > > What is the mechanism for reliably copying exceptions from one > sub-interpreter to another in the `run()` method? If `run()` can raise > an exception, why not let it return values? > > > Cheers, > Mark. > ___ > Python-Dev mailing list -- python-dev@python.org > To unsubscribe send an email to python-dev-le...@python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/ZSE2G37E24YYLNMQKOQSBM46F7KLAOZF/ > Code of Conduct: http://python.org/psf/codeofconduct/ ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/YSZBQEES7LCBANVIRIUXSKDHZGL3Q2F6/ Code of Conduct: http://python.org/psf/codeofconduct/
