[Python-Dev] Re: Static types and subinterpreters running in parallel

[Petr Viktorin]

On 16. 12. 21 20:24, Eric Snow wrote:

On Thu, Dec 16, 2021 at 10:54 AM Guido van Rossum  wrote:


Eric has been looking into this. It's probably the only solution if we can't 
get immutable objects.


Yep.  I've investigated the following approach (for the objects
exposed in the public and limited C-API):

* add a pointer field to PyInterpreterState (or a sub-struct) for each
of the objects
* for the main interpreter, set those pointers to the existing
statically declared objects
* for subinterpreters make a copy (memcpy()?) and fix it up
* add a lookup API and encourage extensions to use it
* for 3.11+ change the symbols to macros:
+ in the internal C-API (Py_BUILD_CORE), the macro would resolve to
the corresponding PyInterpreterState field
+ in the public C-API (and limited API extensions built with
3.11+), the macro would resolve to a call to a (non-inline) lookup
function
+ for limited API extensions built against earlier Python versions
we'd still export the existing symbols
* limited API extensions built against pre-3.11 Python would only be
allowed to run in the main interpreter on 3.11+
+ they probably weren't built with subinterpreters in mind anyway

There are still a number of details to sort out, but nothing that
seems like a huge obstacle.  Here are the ones that come to mind,
along with other details, caveats, and open questions:

* the static types exposed in the C-API are PyObject values rather than pointers
+ I solved this by dereferencing the result of the lookup function
(Guido's idea), e.g. #define PyTuple_Type (*(_Py_GetObject_Tuple()))
* there is definitely a penalty to using a per-interpreter lookup function
+ this would only apply to extension modules since internally we
would access the PyInterpreterState fields directly
+ this is mostly a potential problem only when the object is
directly referenced frequently (e.g. a tight loop),
+ the impact would probably center on use of the high-frequency
singletons (None, True, False) and possibly with Py*_CheckExact()
calls
+ would it be enough of a problem to be worth mitigating?  how
would we do so?
* static types in extensions can't have tp_base set to a builtin type
(since the macro won't resolve)
+ extensions that support subinterpreters (i.e. PEP 489) won't be
using static types (a weak assumption)
+ extensions that do not support subinterpreters and still have
static types would probably break
+ how to fix that?
* limited API extensions built against 3.11+ but running under older
Python versions would break?
+ how to fix that?


With Py_LIMITED_API lower than 3.11, the old API/ABI should continue 
working. But with none of the new features.


Or I guess we could start requiring users to compile stable-ABI 
extensions with the lowest Python version they support (which is 
currently just best practice). That would be a major change, though -- 
and a harsh reminder that the  API is "limited", not "stable" like the ABI.



All in all, immortals do seem like much better solution, if they can be 
made to work.



___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZDDAUMLNLWNTLGRQX3DH3MVAZY35EBVF/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Static types and subinterpreters running in parallel

[Eric Snow]

On Thu, Dec 16, 2021 at 10:54 AM Guido van Rossum  wrote:
>
> Eric has been looking into this. It's probably the only solution if we can't 
> get immutable objects.

Yep.  I've investigated the following approach (for the objects
exposed in the public and limited C-API):

* add a pointer field to PyInterpreterState (or a sub-struct) for each
of the objects
* for the main interpreter, set those pointers to the existing
statically declared objects
* for subinterpreters make a copy (memcpy()?) and fix it up
* add a lookup API and encourage extensions to use it
* for 3.11+ change the symbols to macros:
   + in the internal C-API (Py_BUILD_CORE), the macro would resolve to
the corresponding PyInterpreterState field
   + in the public C-API (and limited API extensions built with
3.11+), the macro would resolve to a call to a (non-inline) lookup
function
   + for limited API extensions built against earlier Python versions
we'd still export the existing symbols
* limited API extensions built against pre-3.11 Python would only be
allowed to run in the main interpreter on 3.11+
   + they probably weren't built with subinterpreters in mind anyway

There are still a number of details to sort out, but nothing that
seems like a huge obstacle.  Here are the ones that come to mind,
along with other details, caveats, and open questions:

* the static types exposed in the C-API are PyObject values rather than pointers
   + I solved this by dereferencing the result of the lookup function
(Guido's idea), e.g. #define PyTuple_Type (*(_Py_GetObject_Tuple()))
* there is definitely a penalty to using a per-interpreter lookup function
   + this would only apply to extension modules since internally we
would access the PyInterpreterState fields directly
   + this is mostly a potential problem only when the object is
directly referenced frequently (e.g. a tight loop),
   + the impact would probably center on use of the high-frequency
singletons (None, True, False) and possibly with Py*_CheckExact()
calls
   + would it be enough of a problem to be worth mitigating?  how
would we do so?
* static types in extensions can't have tp_base set to a builtin type
(since the macro won't resolve)
   + extensions that support subinterpreters (i.e. PEP 489) won't be
using static types (a weak assumption)
   + extensions that do not support subinterpreters and still have
static types would probably break
   + how to fix that?
* limited API extensions built against 3.11+ but running under older
Python versions would break?
   + how to fix that?

> But I would prefer the latter, if we can get the performance penalty low 
> enough.

Absolutely.  Using immortal objects to solve this is a much simpler solution.

-eric
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/7RPTHCLEUHR34PIJKRN453UEWCAI56NW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Static types and subinterpreters running in parallel

[Guido van Rossum]

Eric has been looking into this. It's probably the only solution if we
can't get immutable objects. But I would prefer the latter, if we can get
the performance penalty low enough.

On Thu, Dec 16, 2021 at 2:31 AM Victor Stinner  wrote:

> Hi,
>
> One option to solve the https://bugs.python.org/issue40601 "[C API]
> Hide static types from the limited C API" issue without breaking the
> backward compatibility is to leave the C API and the stable ABI as
> they are for the main interpreter (static types), but force
> subinterpreters running in parallel to use their own heap types. It
> means that subinterpreters would be able to use the main interpreter
> GIL or own their GIL. By default, the GIL would still be shared.
>
> C extensions using "&PyLongType" (static type) would continue to work
> in the main interpreter.
>
> C extensions which want to opt-in for running subinterpreters in
> parallel would not access to "&PyLong_Type" but be forced to call
> PyLong_GetType() (heap type).
>
> Internally, Python should be modified to replace "&PyLongType" with
> PyLong_GetType(). So the code would work with static types and heap
> types.
>
> It also means that a subinterpreter running in parallel would only be
> able to import C extensions built with explicit support for this
> feature. Otherwise, an ImportError would be raised.
>
> Victor
> --
> Night gathers, and now my watch begins. It shall not end until my death.
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/QTY25AHCLOXRCQ2LADUUZFVKNVLLYS25/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/R2R7AHGEWHIRXBQ2VT63E6E5PZGFFZTT/
Code of Conduct: http://python.org/psf/codeofconduct/