[Python-Dev] Re: my plans for subinterpreters (and a per-interpreter GIL)

[Inada Naoki]

On Thu, Jan 6, 2022 at 7:00 AM Trent Nelson  wrote:
>
> I did some research on this a few years back.  I was curious what sort
> of "max reference counts" were encountered in the wild, in long-running
> real life programs.  For the same reason: I wanted to get some insight
> into how many unused bits could possibly be repurposed for future
> shenanigans (I had PyParallel* in the mind at the time).
>

I think we can assume the upper bound of the reference count is same
to upper bound of the pointer.
On 32bit machine, memory space is 2**32 byte, and pointers take
4bytes. And NULL can not store pointer. So upper bound of refcnt is
2**30-1.
So we have two free bits in the refcnt.

On 64bit machine, we have at least four free bits as same reason.

Regards,

-- 
Inada Naoki  
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UNYX6VOQLDMXMWXP54KJUXCHSWOA5YKT/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Sanity check about ctypes

[Gregory P. Smith]

On Wed, Jan 5, 2022 at 3:17 PM Yonatan Zunger  wrote:

> Hey everyone.
>
> Quick sanity check: The ctypes docs
>  refer to
> _CData as a non-public class which is in the module, but _ctypes.c doesn't
> actually export it
> .
> (I discovered this because it turns out that typeshed *is* referencing
> _CData, e.g. in its type annotations for RawIOBase
> 
> )
>
> Is this intended behavior in CPython (in which case the docs are a bit off
> and typeshed has a bug), or is it unexpected to people on this list (in
> which case it's an issue in _ctypes.c)?
>

typeshed is presumably referring to itself. It defines an interface for
ctypes._CData in
https://github.com/python/typeshed/blob/master/stdlib/ctypes/__init__.pyi#L82

The CPython ctypes docs *seem* reasonable to me. There is such a class. It
is not public, so you cannot access ctypes._CData in any direct manner.
That it gets called a class may be somewhat historical - its purpose is to
provide a common interface. What code would ever actually care that it used
class mechanisms as an internal implementation detail to do that?

-gps


>
> Yonatan
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/7L6DNNI3MJ4UIM3C7A7KAIWHX562MRZL/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ENKP334GO73ISSQ5XM5UWTOQYGTROK4E/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: my plans for subinterpreters (and a per-interpreter GIL)

[Trent Nelson]

On Wed, Jan 05, 2022 at 01:59:21PM -0800, Trent Nelson wrote:
> 
> I did some research on this a few years back.  I was curious what sort
> of "max reference counts" were encountered in the wild, in long-running
> real life programs.  For the same reason: I wanted to get some insight
> into how many unused bits could possibly be repurposed for future
> shenanigans (I had PyParallel* in the mind at the time).
> 
> I added some logic to capture* the max reference counts of the None,
> True, and Zero objects (in a trace callback), then ran a really long
> simulation program of a client's (it ran for about 5-6 hours).  The
> results were as follows:
> 
> MaxNoneRefCount 9,364,132
> MaxTrueRefCount   204,215
> MaxZeroRefCount36,784

Just double-checked my results, there were a handful of runs with higher
counts:

MaxNoneRefCount 59,834,444
MaxTrueRefCount  1,072,467
MaxZeroRefCount  3,460,921

Regards,

Trent.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UJPQVBF5I4PGZTBX3EOVLLCAOQVNHVGZ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Function Prototypes

[Brett Cannon]

On Thu, Dec 23, 2021 at 5:13 PM Guido van Rossum  wrote:

> On Thu, Dec 23, 2021 at 3:24 PM Steven D'Aprano 
> wrote:
>
>> On Thu, Dec 23, 2021 at 02:09:50PM -0800, Guido van Rossum wrote:
>>
>> > Without decorator too (that was Lukasz’ idea). Why bother with the
>> > decorator (*if* we were to go there)?
>>
>> So that
>>
>> def func(params): pass
>>
>> creates a function object, and
>>
>> def func(params)
>>
>> makes a Callable type object?
>>
>
> No, no, no. That syntax has already been discredited.
>
> Mark's proposal was
> ```
> @Callable
> def func(params): pass
> ```
> My question is, why does it need `@Callable`?
>

My guess is the thinking is to prevent anyone from thinking that the
function is actually meant to do anything other than be a type signature
for a callable. Otherwise it's very much convention to know what is
purposefully a no-op/always-returns-None function meant to be a callable
type based either on the naming scheme or some other practice like the
ellipsis body versus you just haven't implemented the function yet.


> Lukasz proposed just using any (undecorated) function, with the convention
> being that the body is `...` (to which I would add the convention that the
> function *name* be capitalized, since it is a type). My question (for Mark,
> or for anyone who supports `@Callable`) is why bother with the decorator.
> It should be easy to teach a type checker about this:
>
> ```
> def SomeFn(x: float) -> int:
> ...
>
> def twice(f: SomeFn) -> SomeFn:
> return lambda x: f(f(x))
> ```
>

It probably is more useful without the decorator. For instance, if you want
to type a function parameter to something that acts like `open()`, then
would you want to reproduce that type annotation or just say `opener:
builtins.open`? This would actually lean into duck typing even by allowing
people to easily say, "I want _something_ that directly quacks like this
callable" instead of having to manually duplicate the API with a separate
type annotation (which one could still do if they were worried about code
drift in regards to expectations). Basically it makes all functions and
methods immediately a potential type annotation which could be rather
powerful and could lead to more callables being used compared to
single-method protocols if there isn't some OOP need to pass an object with
a method around.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OWQBEY3ZEWAJAZALSNG47XTEEH53MFLO/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Sanity check about ctypes

[Yonatan Zunger]

Hey everyone.

Quick sanity check: The ctypes docs
 refer to
_CData as a non-public class which is in the module, but _ctypes.c doesn't
actually export it
.
(I discovered this because it turns out that typeshed *is* referencing
_CData, e.g. in its type annotations for RawIOBase

)

Is this intended behavior in CPython (in which case the docs are a bit off
and typeshed has a bug), or is it unexpected to people on this list (in
which case it's an issue in _ctypes.c)?

Yonatan
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/7L6DNNI3MJ4UIM3C7A7KAIWHX562MRZL/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: my plans for subinterpreters (and a per-interpreter GIL)

[Eric Snow]

On Wed, Jan 5, 2022, 15:02 Trent Nelson  wrote:

> I thought that was pretty interesting.  Potentially many, many upper
> bits for the taking.  The code also had some logic that would int 3
> as soon as a 32-bit refcnt overflowed, and that never hit either
> (obviously, based on the numbers above).
>
> I also failed to come up with real-life code that would result in a
> Python object having a reference count higher than None's refcnt, but
> that may have just been from lack of creativity.
>
> Just thought I'd share.
>

Thanks, Trent.  That's super helpful.

-eric

>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/FMSE7AFZVJVBFRQMMYAEAXELITHN2E3B/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: my plans for subinterpreters (and a per-interpreter GIL)

[Trent Nelson]

On Wed, Dec 15, 2021 at 02:57:46PM -0800, Guido van Rossum wrote:
> On Wed, Dec 15, 2021 at 6:04 AM Antoine Pitrou  wrote:
> 
> > On Wed, 15 Dec 2021 14:13:03 +0100
> > Antoine Pitrou  wrote:
> >
> > > Did you try to take into account the envisioned project for adding a
> > > "complete" GC and removing the GIL?
> >
> > Sorry, I was misremembering the details.  Sam Gross' proposal
> > (posted here on 07/10/2021) doesn't switch to a "complete GC", but it
> > changes reference counting to a more sophisticated scheme (which
> > includes immortalization of objects):
> >
> >
> > https://docs.google.com/document/d/18CXhDb1ygxg-YXNBJNzfzZsDFosB5e6BfnXLlejd9l0/edit
> >
> 
> A note about this: Sam's immortalization covers exactly the objects that
> Eric is planning to move into the interpreter state struct: "such as
> interned strings, small integers, statically allocated PyTypeObjects, and
> the True, False, and None objects". (Well, he says "such as" but I think so
> does Eric. :-)
> 
> Sam's approach is to use the lower bit of the ob_refcnt field to indicate
> immortal objects. This would not work given the stable ABI (which has
> macros that directly increment and decrement the ob_refcnt field). In fact,
> I think that Sam's work doesn't preserve the stable ABI at all. However,
> setting a very high bit (the bit just below the sign bit) would probably
> work. Say we're using 32 bits. We use the value 0x_6000_ as the initial
> refcount for immortal objects. The stable ABI will sometimes increment
> this, sometimes decrement it. But as long as the imbalance is less than
> 0x_2000_, the refcount will remain in the inclusive range [
> 0x_4000_ , 0x_7FFF_ ] and we can test for immortality by testing a
> single bit:
> 
> if (o->ob_refcnt & 0x_4000_)
> 
> I don't know how long that would take, but I suspect that a program that
> just increments the refcount relentlessly would have to run for hours
> before hitting this range. On a 64-bit machine the same approach would
> require years to run before a refcount would exceed the maximum allowable
> imbalance. (These estimates are from Mark Shannon.)

I did some research on this a few years back.  I was curious what sort
of "max reference counts" were encountered in the wild, in long-running
real life programs.  For the same reason: I wanted to get some insight
into how many unused bits could possibly be repurposed for future
shenanigans (I had PyParallel* in the mind at the time).

I added some logic to capture* the max reference counts of the None,
True, and Zero objects (in a trace callback), then ran a really long
simulation program of a client's (it ran for about 5-6 hours).  The
results were as follows:

MaxNoneRefCount 9,364,132
MaxTrueRefCount   204,215
MaxZeroRefCount36,784

I thought that was pretty interesting.  Potentially many, many upper
bits for the taking.  The code also had some logic that would int 3
as soon as a 32-bit refcnt overflowed, and that never hit either
(obviously, based on the numbers above).

I also failed to come up with real-life code that would result in a
Python object having a reference count higher than None's refcnt, but
that may have just been from lack of creativity.

Just thought I'd share.

Regards,

Trent.

[*] 1: https://github.com/pyparallel/pyparallel
[*] 2: 
https://github.com/tpn/tracer/blob/master/PythonTracer/PythonTracer.h#L690


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/U2WHO5LYMBF6A6AFM36HOQQCNVGLXG6M/
Code of Conduct: http://python.org/psf/codeofconduct/