[Python-Dev] Re: "immortal" objects and how they would help per-interpreter GIL

On Fri, Dec 17, 2021 at 11:35:24AM +1300, Greg Ewing wrote: > On 17/12/21 6:52 am, Eddie Elizondo via Python-Dev wrote: > >I've just updated the original Immortal Instances PR > > Is it just me, or does Immortal Instances sound like a > video game franchise? Or a Doctor Who episode. Doctor Who

[Python-Dev] Re: subinterpreters and their possible impact on large extension projects

Petr Viktorin wrote: >>> In Python 3.11, Python still implements around 100 types as "static >>> types" which are not compatible with subinterpreters, ... >>> seems like changing it may break the C API *and* the stable ABI > > If sub-interpreters each need their own copy of even immutable

[Python-Dev] Re: "immortal" objects and how they would help per-interpreter GIL

On 17/12/21 6:52 am, Eddie Elizondo via Python-Dev wrote: I've just updated the original Immortal Instances PR Is it just me, or does Immortal Instances sound like a video game franchise? -- Greg ___ Python-Dev mailing list -- python-dev@python.org

[Python-Dev] Re: "immortal" objects and how they would help per-interpreter GIL

It’s *needed* when multiple interpreters share them. On Thu, Dec 16, 2021 at 14:03 Jim J. Jewett wrote: > Guido van Rossum wrote: > > On Wed, Dec 15, 2021 at 6:57 PM Jim J. Jewett jimjjew...@gmail.com > wrote: > > > Immortal objects shouldn't be reclaimed by garbage collection, but they > > >

[Python-Dev] Re: "immortal" objects and how they would help per-interpreter GIL

Guido van Rossum wrote: > On Wed, Dec 15, 2021 at 6:57 PM Jim J. Jewett jimjjew...@gmail.com wrote: > > Immortal objects shouldn't be reclaimed by garbage collection, but they > > still count as potential external roots for non-cyclic liveness. > So everything referenced by an immortal object

[Python-Dev] Re: Static types and subinterpreters running in parallel

On Thu, Dec 16, 2021 at 10:54 AM Guido van Rossum wrote: > > Eric has been looking into this. It's probably the only solution if we can't > get immutable objects. Yep. I've investigated the following approach (for the objects exposed in the public and limited C-API): * add a pointer field to

[Python-Dev] Re: subinterpreters and their possible impact on large extension projects

On Thu, 16 Dec 2021 11:38:28 -0700 Eric Snow wrote: > On Thu, Dec 16, 2021 at 4:34 AM Antoine Pitrou wrote: > > As a data point, in PyArrow, we have a bunch of C++ code that interacts > > with Python but doesn't belong in a particular Python module. That C++ > > code can of course have global

[Python-Dev] Re: subinterpreters and their possible impact on large extension projects

On Thu, Dec 16, 2021 at 4:34 AM Antoine Pitrou wrote: > As a data point, in PyArrow, we have a bunch of C++ code that interacts > with Python but doesn't belong in a particular Python module. That C++ > code can of course have global state, including perhaps Python objects. Thanks for that

[Python-Dev] Re: "immortal" objects and how they would help per-interpreter GIL

On Thu, Dec 16, 2021 at 3:08 AM Petr Viktorin wrote: > On 16. 12. 21 11:52, Victor Stinner wrote: > > On Thu, Dec 16, 2021 at 2:29 AM Nathaniel Smith wrote: > >> On Wed, Dec 15, 2021 at 3:07 AM Victor Stinner > wrote: > >>> I wrote https://bugs.python.org/issue39511 and > >>>

[Python-Dev] Re: Static types and subinterpreters running in parallel

Eric has been looking into this. It's probably the only solution if we can't get immutable objects. But I would prefer the latter, if we can get the performance penalty low enough. On Thu, Dec 16, 2021 at 2:31 AM Victor Stinner wrote: > Hi, > > One option to solve the

[Python-Dev] RFC on Callable Syntax PEP

Hello all, Thanks everyone for comments on our earlier thread [1] about callable type syntax. We now have a draft PEP [2] proposing an arrow-based syntax for callable types, for example: ``` (int, str) -> bool # equivalent to Callable[[int, str], bool] ``` In support of the PEP we also

[Python-Dev] Re: "immortal" objects and how they would help per-interpreter GIL

I've just updated the original Immortal Instances PR with a bunch of tricks that I used to achieve as much performance parity as possible: https://github.com/python/cpython/pull/19474 . You can see the details along with some benchmarks in the PR itself. This should address a bunch of the

[Python-Dev] Re: my plans for subinterpreters (and a per-interpreter GIL)

On Thu, Dec 16, 2021 at 2:48 AM Petr Viktorin wrote: > But does the sign bit need to stay intact, and do we actually need to > rely on the immortal bit to always be set for immortal objects? > If the refcount rolls over to zero, an immortal object's dealloc could > bump it back and give itself

[Python-Dev] Re: my plans for subinterpreters (and a per-interpreter GIL)

On Thu, Dec 16, 2021 at 01:46:38PM +0100, Antoine Pitrou wrote: > If an object is immortal, then its refcount wouldn't change at all. Ah, that makes sense, thanks for the explanation. -- Steve ___ Python-Dev mailing list -- python-dev@python.org To

[Python-Dev] Re: subinterpreters and their possible impact on large extension projects

On Thu, 16 Dec 2021 13:25:53 +0100 Petr Viktorin wrote: > On 16. 12. 21 12:33, Antoine Pitrou wrote: > > On Tue, 14 Dec 2021 10:38:25 -0700 > > Eric Snow wrote: > >> > >> So we (the core devs) would effectively be requiring those extensions > >> to support subinterpreters, regardless of

[Python-Dev] Re: my plans for subinterpreters (and a per-interpreter GIL)

On Thu, 16 Dec 2021 23:32:17 +1100 Steven D'Aprano wrote: > On Thu, Dec 16, 2021 at 12:23:09PM +0100, Antoine Pitrou wrote: > > > > The "real number of references" would not be known for immortal objects. > > Oh that surprises me. How does that work? Does that imply that some code > might

[Python-Dev] Re: my plans for subinterpreters (and a per-interpreter GIL)

On Thu, Dec 16, 2021 at 12:23:09PM +0100, Antoine Pitrou wrote: > > The "real number of references" would not be known for immortal objects. Oh that surprises me. How does that work? Does that imply that some code might not increment the ref count, while other code will? -- Steve

[Python-Dev] Re: subinterpreters and their possible impact on large extension projects

On 16. 12. 21 12:33, Antoine Pitrou wrote: On Tue, 14 Dec 2021 10:38:25 -0700 Eric Snow wrote: So we (the core devs) would effectively be requiring those extensions to support subinterpreters, regardless of letting them opt out. This situation has been weighing heavily on my mind since

[Python-Dev] Re: subinterpreters and their possible impact on large extension projects

On Tue, 14 Dec 2021 10:38:25 -0700 Eric Snow wrote: > > So we (the core devs) would effectively be requiring those extensions > to support subinterpreters, regardless of letting them opt out. This > situation has been weighing heavily on my mind since Nathaniel brought > this up. Here are some

[Python-Dev] Re: my plans for subinterpreters (and a per-interpreter GIL)

On Thu, 16 Dec 2021 14:32:05 +1100 Steven D'Aprano wrote: > On Wed, Dec 15, 2021 at 02:57:46PM -0800, Guido van Rossum wrote: > > > Another potential issue is that there may be some applications that take > > refcounts at face value (perhaps obtained using sys.getrefcount()). These > > would

[Python-Dev] Re: my plans for subinterpreters (and a per-interpreter GIL)

On Thu, Dec 16, 2021 at 12:00 AM Guido van Rossum wrote: > Sam's approach is to use the lower bit of the ob_refcnt field to indicate > immortal objects. This would not work given the stable ABI (which has macros > that directly increment and decrement the ob_refcnt field). (...) we can test >

[Python-Dev] Re: "immortal" objects and how they would help per-interpreter GIL

On 16. 12. 21 11:52, Victor Stinner wrote: On Thu, Dec 16, 2021 at 2:29 AM Nathaniel Smith wrote: On Wed, Dec 15, 2021 at 3:07 AM Victor Stinner wrote: I wrote https://bugs.python.org/issue39511 and https://github.com/python/cpython/pull/18301 to have per-interpreter None, True and False

[Python-Dev] Re: "immortal" objects and how they would help per-interpreter GIL

On Thu, Dec 16, 2021 at 2:29 AM Nathaniel Smith wrote: > On Wed, Dec 15, 2021 at 3:07 AM Victor Stinner wrote: > > I wrote https://bugs.python.org/issue39511 and > > https://github.com/python/cpython/pull/18301 to have per-interpreter > > None, True and False singletons. My change is backward

[Python-Dev] Static types and subinterpreters running in parallel

Hi, One option to solve the https://bugs.python.org/issue40601 "[C API] Hide static types from the limited C API" issue without breaking the backward compatibility is to leave the C API and the stable ABI as they are for the main interpreter (static types), but force subinterpreters running in

[Python-Dev] Re: subinterpreters and their possible impact on large extension projects

On 16. 12. 21 3:41, Jim J. Jewett wrote: In Python 3.11, Python still implements around 100 types as "static types" which are not compatible with subinterpreters, like _Type and _Type. I opened https://bugs.python.org/issue40601 about these static types, but it seems like changing it may break

[Python-Dev] Re: my plans for subinterpreters (and a per-interpreter GIL)

On 16. 12. 21 2:54, Guido van Rossum wrote: (I just realized that we started discussing details of immortal objects in the wrong thread -- this is Eric's overview thread, there's a separate thread on immortal objects. But alla, I'll respond here below.) On Wed, Dec 15, 2021 at 5:05 PM Neil

[Python-Dev] Re: my plans for subinterpreters (and a per-interpreter GIL)

On 15. 12. 21 23:57, Guido van Rossum wrote: On Wed, Dec 15, 2021 at 6:04 AM Antoine Pitrou > wrote: On Wed, 15 Dec 2021 14:13:03 +0100 Antoine Pitrou mailto:anto...@python.org>> wrote: > Did you try to take into account the envisioned project for adding