> On 21 Apr 2020, at 16:58, Eric Snow <ericsnowcurren...@gmail.com> wrote:
>
> Thanks for explaining that, Ronald. It sounds like a lot of the
> effort would relate to making classes work. I have some comments
> in-line below.
>
> -eric
>
> On Tue, Apr 21, 2020 at 2:34 AM Ronald Oussoren <ronaldousso...@mac.com>
> wrote:
>>> On 21 Apr 2020, at 03:21, Eric Snow <ericsnowcurren...@gmail.com> wrote:
>>> Honest question: how many C extensions have process-global state that
>>> will cause problems under subinterpreters? In other words, how many
>>> already break in mod_wsgi?
>>
>> Fully supporting sub-interpreters in PyObjC will likely be a lot of work,
>> mostly
>> due to being able to subclass Objective-C classes from Python. With sub-
>> interpreters a Python script in an interpreter could see an Objective-C
>> class in
>> a different sub-interpreter. The current PyObjC architecture assumes that
>> there’s exactly one (sub-)interpreter, that’s probably fixable but is far
>> from trivial.
>
> Are the Objective-C classes immutable? Are the wrappers stateful at
> all?
> Without context I'm not clear on how you would be impacted by
> operation under subinterpreters (i.e. PEP 554), but it sounds like the
> classes do have global state that is in fact interpreter-specific. I
> expect you would also be impacted by subinterpreters not sharing the
> GIL but that is a separate matter (see below).
My mail left out some important information, sorry about that.
PyObjC is a two-way bridge between Python and Objective-C. One half of this is
that is bridging Objective-C classes (and instances) to Python. This is fairly
straightforward, although the proxy objects are not static and can have methods
defined in Python (helper methods that make the Objective-C classes nicer to
use from Python, for example to define methods that make it possible to use an
NSDictionary as if it were a regular Python dict).
The other half is that it is possible to implement Objective-C classes in
Python:
class MyClass (Cocoa.NSObject):
def anAction_(self, sender): …
This defines a Python classes named “MyClass”, but also an Objective-C class of
the same name that forwards Objective-C calls to Python. The implementation
for this uses PyGILState_Ensure, which AFAIK is not yet useable with
sub-interpreters.
PyObjC also has Objective-C proxy classes for generic Python objects, making it
possible to pass a normal Python dictionary to an Objective-C API that expects
an NSDictionary instance.
Things get interesting when combining the two with sub-interpreters: With the
current implementation the Objective-C world would be a channel for passing
“live” Python objects between sub-interpreters.
The translation tables for looking up existing proxies (mapping from Python to
Objective-C and vice versa) are currently singletons.
This is probably fixable with another level of administration, by keeping track
of the sub-interpreter that owns a Python object I could ensure that Python
objects owned by a different sub-interpreter are proxied like any other
Objective-C object which would close this loophole. That would require
significant changes to a code base that’s already fairly complex, but should be
fairly straightforward.
>
> Regardless, I expect there are others in a similar situation. It
> would be good to understand your use case and help with a solution.
> Is there a specific example you can point to of code that would be
> problematic under subinterpreters?
>
>> With the current API it might not even be possible to add sub-interpreter
>> support
>
> What additional API would be needed?
See above, the main problem is PyGILState_Ensure. I haven’t spent a lot of
time thinking about this though, I might find other issues when I try to
support sub-interpreters.
>
>> (although I write this without having read the PEP).
>
> Currently PEP 554 does not talk about how to make extension modules
> compatible with subinterpreters. That may be worth doing, though it
> would definitely have to happen in the docs and (to an extent) the 3.9
> "What's New" page. There is already some discussion on what should be
> in those docs (see
> https://github.com/ericsnowcurrently/multi-core-python/issues/53).
>
> Note that, until the GIL becomes per-interpreter, sharing objects
> isn't a problem. We were not planning on having a PEP for the
> stop-sharing-the-GIL effort, but I'm starting to think that it may be
> worth it, to cover the impact on extension modules (e.g. mitigations).
>
> So if you leave out the complications due to not sharing the GIL, the
> main problem extension authors face with subinterpreters (exposed by
> PEP 554) is when their module has process-global state that breaks
> under subinterpreters. From your description above, it sounds like
> you may be in that situation.
>
>> As far as I understand proper support for subinterpreters also requires
>> moving
>> away from static type definitions to avoid sharing objects between
>> interpreters
>> (that is, use the PyType_FromSpec to build types).
>
> Correct, though that is not technically a problem until we stop sharing the
> GIL.
Right. But a major selling point of sub-interpreters is that this provide a way
forward towards having multiple Python threads that don’t share a GIL.
IMHO it would be better to first work out what’s needed to get there, and in
particular what changes are needed in extensions. Otherwise extensions may have
to be changed multiple times.
>
>> At first glance this API does not support everything I do in PyObjC (fun
>> with metaclasses, in C code).
>
> What specific additions/changes would you need?
At least:
- A variant of PyGILState_Ensure that supports sub-interpreters
- Defining subclasses of built-in types using PyType_FromSpec, in particular a
subclass of “type”.
BTW. In my first mail I mentioned I don’t have a use cases for subinterpreters.
I might have a limited use case in the PyObjC domain: implementing plugin
bundles for Objective-C applications in Python. These currently share the same
interpreter, which can cause problems. Subinterpreters could be helpful there
to isolate code, but that would require having an API that conditionally
initialises the Python runtime (similar to PyGILState_Ensure, but for the
runtime itself). This wouldn’t fix all problems because you can’t have two
different python versions in one proces, but would be better than the status
quo.
Ronald
—
Twitter / micro.blog: @ronaldoussoren
Blog: https://blog.ronaldoussoren.net/
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/UAGGZSDKGRQ5N64OGKN4QCJUVDUZIODM/
Code of Conduct: http://python.org/psf/codeofconduct/
