[Python-Dev] Re: my plans for subinterpreters (and a per-interpreter GIL)

On Thu, Jan 6, 2022 at 7:00 AM Trent Nelson wrote: > > I did some research on this a few years back. I was curious what sort > of "max reference counts" were encountered in the wild, in long-running > real life programs. For the same reason: I wanted to get some insight > into how many unused

[Python-Dev] Re: my plans for subinterpreters (and a per-interpreter GIL)

On Wed, Jan 05, 2022 at 01:59:21PM -0800, Trent Nelson wrote: > > I did some research on this a few years back. I was curious what sort > of "max reference counts" were encountered in the wild, in long-running > real life programs. For the same reason: I wanted to get some insight > into how

[Python-Dev] Re: my plans for subinterpreters (and a per-interpreter GIL)

On Wed, Jan 5, 2022, 15:02 Trent Nelson wrote: > I thought that was pretty interesting. Potentially many, many upper > bits for the taking. The code also had some logic that would int 3 > as soon as a 32-bit refcnt overflowed, and that never hit either > (obviously, based on the numbers

[Python-Dev] Re: my plans for subinterpreters (and a per-interpreter GIL)

On Wed, Dec 15, 2021 at 02:57:46PM -0800, Guido van Rossum wrote: > On Wed, Dec 15, 2021 at 6:04 AM Antoine Pitrou wrote: > > > On Wed, 15 Dec 2021 14:13:03 +0100 > > Antoine Pitrou wrote: > > > > > Did you try to take into account the envisioned project for adding a > > > "complete" GC and

[Python-Dev] Re: my plans for subinterpreters (and a per-interpreter GIL)

On Wed, 15 Dec 2021, 3:18 am Eric Snow, wrote: > Hi all, > > I'm still hoping to land a per-interpreter GIL for 3.11. There is > still a decent amount of work to be done but little of it will require > solving any big problems: > > * pull remaining static globals into _PyRuntimeState and

[Python-Dev] Re: my plans for subinterpreters (and a per-interpreter GIL)

On Thu, Dec 16, 2021 at 2:48 AM Petr Viktorin wrote: > But does the sign bit need to stay intact, and do we actually need to > rely on the immortal bit to always be set for immortal objects? > If the refcount rolls over to zero, an immortal object's dealloc could > bump it back and give itself

[Python-Dev] Re: my plans for subinterpreters (and a per-interpreter GIL)

On Thu, Dec 16, 2021 at 01:46:38PM +0100, Antoine Pitrou wrote: > If an object is immortal, then its refcount wouldn't change at all. Ah, that makes sense, thanks for the explanation. -- Steve ___ Python-Dev mailing list -- python-dev@python.org To

[Python-Dev] Re: my plans for subinterpreters (and a per-interpreter GIL)

On Thu, 16 Dec 2021 23:32:17 +1100 Steven D'Aprano wrote: > On Thu, Dec 16, 2021 at 12:23:09PM +0100, Antoine Pitrou wrote: > > > > The "real number of references" would not be known for immortal objects. > > Oh that surprises me. How does that work? Does that imply that some code > might

[Python-Dev] Re: my plans for subinterpreters (and a per-interpreter GIL)

On Thu, Dec 16, 2021 at 12:23:09PM +0100, Antoine Pitrou wrote: > > The "real number of references" would not be known for immortal objects. Oh that surprises me. How does that work? Does that imply that some code might not increment the ref count, while other code will? -- Steve

[Python-Dev] Re: my plans for subinterpreters (and a per-interpreter GIL)

On Thu, 16 Dec 2021 14:32:05 +1100 Steven D'Aprano wrote: > On Wed, Dec 15, 2021 at 02:57:46PM -0800, Guido van Rossum wrote: > > > Another potential issue is that there may be some applications that take > > refcounts at face value (perhaps obtained using sys.getrefcount()). These > > would

[Python-Dev] Re: my plans for subinterpreters (and a per-interpreter GIL)

On Thu, Dec 16, 2021 at 12:00 AM Guido van Rossum wrote: > Sam's approach is to use the lower bit of the ob_refcnt field to indicate > immortal objects. This would not work given the stable ABI (which has macros > that directly increment and decrement the ob_refcnt field). (...) we can test >

[Python-Dev] Re: my plans for subinterpreters (and a per-interpreter GIL)

On 16. 12. 21 2:54, Guido van Rossum wrote: (I just realized that we started discussing details of immortal objects in the wrong thread -- this is Eric's overview thread, there's a separate thread on immortal objects. But alla, I'll respond here below.) On Wed, Dec 15, 2021 at 5:05 PM Neil

[Python-Dev] Re: my plans for subinterpreters (and a per-interpreter GIL)

On 15. 12. 21 23:57, Guido van Rossum wrote: On Wed, Dec 15, 2021 at 6:04 AM Antoine Pitrou > wrote: On Wed, 15 Dec 2021 14:13:03 +0100 Antoine Pitrou mailto:anto...@python.org>> wrote: > Did you try to take into account the envisioned project for adding

[Python-Dev] Re: my plans for subinterpreters (and a per-interpreter GIL)

On Wed, Dec 15, 2021 at 02:57:46PM -0800, Guido van Rossum wrote: > Another potential issue is that there may be some applications that take > refcounts at face value (perhaps obtained using sys.getrefcount()). These > would find that immortal objects have a very large refcount, which might >

[Python-Dev] Re: my plans for subinterpreters (and a per-interpreter GIL)

On Wed, Dec 15, 2021 at 6:21 PM Guido van Rossum wrote: > On Wed, Dec 15, 2021 at 2:57 PM Guido van Rossum wrote: > >> >> I don't know how long that would take, but I suspect that a program that >> just increments the refcount relentlessly would have to run for hours >> before hitting this

[Python-Dev] Re: my plans for subinterpreters (and a per-interpreter GIL)

On Wed, Dec 15, 2021 at 2:57 PM Guido van Rossum wrote: > > I don't know how long that would take, but I suspect that a program that > just increments the refcount relentlessly would have to run for hours > before hitting this range. On a 64-bit machine the same approach would > require years to

[Python-Dev] Re: my plans for subinterpreters (and a per-interpreter GIL)

(I just realized that we started discussing details of immortal objects in the wrong thread -- this is Eric's overview thread, there's a separate thread on immortal objects. But alla, I'll respond here below.) On Wed, Dec 15, 2021 at 5:05 PM Neil Schemenauer wrote: > On 2021-12-15 2:57 p.m.,

[Python-Dev] Re: my plans for subinterpreters (and a per-interpreter GIL)

On 2021-12-15 2:57 p.m., Guido van Rossum wrote: But as long as the imbalance is less than 0x_2000_, the refcount will remain in the inclusive range [ 0x_4000_ , 0x_7FFF_ ] and we can test for immortality by testing a single bit: if (o->ob_refcnt & 0x_4000_) Could we have a

[Python-Dev] Re: my plans for subinterpreters (and a per-interpreter GIL)

>> It does seem a bit silly to actually be tracking that refcount :-) Not that silly. It can easily help in C extensions to detect wrong DECREF calls: >> import ctypes >> non = ctypes.c_long.from_address(id(None)) >> non.value = 10 >> Fatal Python error: none_dealloc: deallocating None Python

[Python-Dev] Re: my plans for subinterpreters (and a per-interpreter GIL)

On Wed, Dec 15, 2021 at 3:00 PM Guido van Rossum wrote: > who cares if the refcount for None is 5000 or 1610612736? As long as the > refcount of *mortal* objects is the same as it was before, this shouldn't > be a problem. > indeed: $ python -c "import sys; print(sys.getrefcount(None))" 4110

[Python-Dev] Re: my plans for subinterpreters (and a per-interpreter GIL)

On Tue, Dec 14, 2021 at 10:12 AM Eric Snow wrote: > * it is fully backward compatible and the C-API is essentially unaffected Hmm, this is a little misleading. It will definitely be backward incompatible for extension modules that don't work under multiple subinterpreters (or rely on the GIL to

[Python-Dev] Re: my plans for subinterpreters (and a per-interpreter GIL)

On Wed, Dec 15, 2021 at 6:04 AM Antoine Pitrou wrote: > On Wed, 15 Dec 2021 14:13:03 +0100 > Antoine Pitrou wrote: > > > Did you try to take into account the envisioned project for adding a > > "complete" GC and removing the GIL? > > Sorry, I was misremembering the details. Sam Gross' proposal

[Python-Dev] Re: my plans for subinterpreters (and a per-interpreter GIL)

On Wed, Dec 15, 2021 at 6:16 AM Antoine Pitrou wrote: > Did you try to take into account the envisioned project for adding a > "complete" GC and removing the GIL? Yeah. I was going to start a separate thread about per-interpreter GIL vs. no-gil, but figured I was already pushing my luck with 3

[Python-Dev] Re: my plans for subinterpreters (and a per-interpreter GIL)

On Wed, 15 Dec 2021 14:13:03 +0100 Antoine Pitrou wrote: > Hi Eric, > > Did you try to take into account the envisioned project for adding a > "complete" GC and removing the GIL? Sorry, I was misremembering the details. Sam Gross' proposal (posted here on 07/10/2021) doesn't switch to a

[Python-Dev] Re: my plans for subinterpreters (and a per-interpreter GIL)

Hi Eric, Did you try to take into account the envisioned project for adding a "complete" GC and removing the GIL? Regards Antoine. On Tue, 14 Dec 2021 10:12:07 -0700 Eric Snow wrote: > Hi all, > > I'm still hoping to land a per-interpreter GIL for 3.11. There is > still a decent amount

[Python-Dev] Re: my plans for subinterpreters (and a per-interpreter GIL)

+1 for consolidated documentation about per-interpreter GIL. On Tue, Dec 14, 2021 at 9:12 AM Eric Snow wrote: > Hi all, > > I'm still hoping to land a per-interpreter GIL for 3.11. There is > still a decent amount of work to be done but little of it will require > solving any big problems: > >

[Python-Dev] Re: my plans for subinterpreters (and a per-interpreter GIL)

Hi, On Tue, Dec 14, 2021 at 6:13 PM Eric Snow wrote: > I'm still hoping to land a per-interpreter GIL for 3.11. There is > still a decent amount of work to be done but little of it will require > solving any big problems: > > * pull remaining static globals into _PyRuntimeState and

[Python-Dev] Re: my plans for subinterpreters (and a per-interpreter GIL)

Whoops, never mind, I see that you started the "immortal objects" thread to discuss this. On Tue, Dec 14, 2021 at 4:54 PM Nathaniel Smith wrote: > > How did you end up solving the issue where Py_None is a static global > that's exposed as part of the stable C ABI? > > On Tue, Dec 14, 2021 at

[Python-Dev] Re: my plans for subinterpreters (and a per-interpreter GIL)

How did you end up solving the issue where Py_None is a static global that's exposed as part of the stable C ABI? On Tue, Dec 14, 2021 at 9:13 AM Eric Snow wrote: > > Hi all, > > I'm still hoping to land a per-interpreter GIL for 3.11. There is > still a decent amount of work to be done but