[issue43503] [subinterpreters] PyObject statics exposed in the limited API break isolation.

[Eric Snow]

Eric Snow  added the comment:

FYI, I'm going to focus discussion on the capi-sig thread.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43503] [subinterpreters] PyObject statics exposed in the limited API break isolation.

[Eric Snow]

Eric Snow  added the comment:

> PEP 489 is *very much* part of the limited API.

Gah, I missed that.  That said, I don't think it matters; I just lose an easy 
point in the rationale. :)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43503] [subinterpreters] PyObject statics exposed in the limited API break isolation.

[Eric Snow]

Eric Snow  added the comment:

> I am confused. How can widening the usable number of functions (i.e. using
> the whole C-API rather than the limited API) help c-extension modules be
> usable in subinterpreters? Aren't the same singletons, exception types, and
> other types exposed in the full C-API?

If Py_LIMITED_API is defined then things would stay the same.  Otherwise we 
would replace the names with macros to do the appropriate lookup.  (That isn't 
the whole story since the Py*_Type names are PyTypeObject and not PyObject*.)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43503] [subinterpreters] PyObject statics exposed in the limited API break isolation.

[STINNER Victor]

STINNER Victor  added the comment:

Eric Snow proposes that C extensions which want to be compatible with 
subinterpreters must use an hypothetical variant of the C API which doesn't 
inherit flaws of the current C API. For example, static types like 
"_Type" would be excluded.

To be clear, the limited C API does expose (indirectly) "_Type". We are 
talking about a new variant of the C API.

The main interpreter would continue to use its static type "_Type", 
whereas each subinterpreter would get its own "int" type allocated on the heap 
(heap type).

Someone has to write a PoC to ensure that this idea works in practice.

In bpo-40601, I proposed that all interpreters including the main interpreter 
only use heap types: remove "_Type" from the C API which is a backward 
incompatible C API change.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43503] [subinterpreters] PyObject statics exposed in the limited API break isolation.

[Petr Viktorin]

Petr Viktorin  added the comment:

There seems to be much confusion here. Maybe on my side?

PEP 489 is *very much* part of the limited API.

--
nosy: +petr.viktorin

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43503] [subinterpreters] PyObject statics exposed in the limited API break isolation.

[mattip]

mattip  added the comment:

I am confused. How can widening the usable number of functions (i.e. using the 
whole C-API rather than the limited API) help c-extension modules be usable in 
subinterpreters? Aren't the same singletons, exception types, and other types 
exposed in the full C-API?

--
nosy: +mattip

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43503] [subinterpreters] PyObject statics exposed in the limited API break isolation.

[Eric Snow]

Eric Snow  added the comment:

FYI, I posted to capi-sig about this:

https://mail.python.org/archives/list/capi-...@python.org/thread/INLCGPMTYFLRTWQL7RB4MUQZ37JAFRAU/

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43503] [subinterpreters] PyObject statics exposed in the limited API break isolation.

[Eric Snow]

Eric Snow  added the comment:

One simple solution is to explicitly state that the limited API does not 
support subinterpreters.  This is already implied by the fact that the 
multi-phase init API (PEP 489) requires subinterpreter support but is not part 
of the limited API.

If we establish this constraint then the problem I originally described here 
goes away (and we can close this issue).

(Note: I'm pretty sure this is something someone suggested to me at some point, 
rather than my own idea.)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43503] [subinterpreters] PyObject statics exposed in the limited API break isolation.

[STINNER Victor]

STINNER Victor  added the comment:

> I can never remember what "Py_LIMITED_API" stands for.

Include/README file is being written, have a look ;-)
https://github.com/python/cpython/pull/24884/files

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43503] [subinterpreters] PyObject statics exposed in the limited API break isolation.

[STINNER Victor]

STINNER Victor  added the comment:

> * 5 singletons

This issue is discussed in bpo-39511 "[subinterpreters] Per-interpreter 
singletons (None, True, False, etc.)".

> Since they are part of the limited API, they have a direct effect on the 
> stable ABI.

This issue is discussed in bpo-40601: "[C API] Hide static types from the 
limited C API".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43503] [subinterpreters] PyObject statics exposed in the limited API break isolation.

[Guido van Rossum]

Guido van Rossum  added the comment:

I can never remember what "Py_LIMITED_API" stands for. If it's not defined, 
does that mean we have the *unlimited* API? Is that a superset or a subset of 
the limited API?

Regarding 1a *and* 1b, I think it would help to list the specific reasons 
exceptions and other types are not entirely immutable. Is it just 
__subclasses__ or is there other state (apart from the refcount) that's mutable 
and visible to the end user? (Or even if it's visible to C API users.)

--
nosy: +gvanrossum

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43503] [subinterpreters] PyObject statics exposed in the limited API break isolation.

[Eric Snow]

Change by Eric Snow :


--
components: +Extension Modules, Interpreter Core, Subinterpreters

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43503] [subinterpreters] PyObject statics exposed in the limited API break isolation.

[Eric Snow]

Change by Eric Snow :


--
keywords: +patch
pull_requests: +23642
stage: needs patch -> patch review
pull_request: https://github.com/python/cpython/pull/24828

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43503] [subinterpreters] PyObject statics exposed in the limited API break isolation.

[Eric Snow]

Change by Eric Snow :


--
nosy: +Mark.Shannon, nascheme, vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43503] [subinterpreters] PyObject statics exposed in the limited API break isolation.

[Eric Snow]

Eric Snow  added the comment:

If the stable ABI weren't an issue then we would probably:

* deprecate using the objects directly
* do something like (2a) in the meantime

It may make sense to do that for "#ifndef Py_LIMITED_API", regardless of how we 
handle the limited API.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43503] [subinterpreters] PyObject statics exposed in the limited API break isolation.

[Eric Snow]

Eric Snow  added the comment:

Here are some solutions that I've considered:

1. immutable objects
   a. make the objects truly immutable/const
  * not trivial, if possible
   b. make the objects effectively immutable
  * (see GH-24828) use a really high refcount to make races irrelevant
2. per-interpreter objects
   a. replace them with macros that do a per-interpreter lookup
   b. replace them with simple placeholders and do a per-interpreter lookup 
internally
   c. replace them with PyObject placeholders and do a per-interpreter lookup 
internally

As far as I'm aware, only (1b) and (2c) are realistic and won't break the 
stable ABI (i.e. preserve layout).

(FWIW, I think that even with (1b) we would still have per-interpreter objects.)

-- Regarding (1a) --

See see GH-24828 for an example implementation.  This includes storing some 
state for the objects in PyInterpreterState and doing a lookup internally.

pros:
* relatively straightforward to implement
* overlaps with other interests (see bpo-40255)
* makes the objects shareable between interpreters (making them more efficient)

cons:
* we have to ensure the objects stay immutable (a tractable problem if the 
solution is constrained to only the limited API objects)

-- Regarding (2c) --

This involves adding an API to get the per-interpreter object for a given 
identity value (flag, global, etc.) and then mapping the limited API objects to 
the corresponding per-interpreter ones.

pros:
* avoids a one-off solution
* extensions can stop using the variables directly (in favor of the new lookup 
API)

cons:
* effectively couples the C-API to the design (as long as the objects are in 
the limited API)
* many touches all over the C-API
* any future changes/additions in the C-API must deal with the objects

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43503] [subinterpreters] PyObject statics exposed in the limited API break isolation.

[Eric Snow]

New submission from Eric Snow :

In the limited C-API we expose the following static PyObject variables:

* 5 singletons
* ~70 exception types
* ~70 other types

Since they are part of the limited API, they have a direct effect on the stable 
ABI.

The problem is that these objects should not be shared between interpreters.  
There are a number of possible solutions for isolating the objects, but the big 
constraint is that the solution cannot break the stable ABI.

--
components: C API
messages: 388759
nosy: eric.snow
priority: normal
severity: normal
stage: needs patch
status: open
title: [subinterpreters] PyObject statics exposed in the limited API break 
isolation.
type: behavior
versions: Python 3.10

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com