[Python-Dev] Re: Proto-PEP part 4: The wonderful third option

[Paul Bryan]

Thanks, Carl and Larry for the explanations.

On Sun, 2022-05-01 at 16:13 -0600, Carl Meyer wrote:
> Hi Paul,
> 
> On Sun, May 1, 2022 at 3:47 PM Paul Bryan  wrote:
> > 
> > Can someone state what's currently unpalatable about 649? It seemed
> > to address the forward-referencing issues, certainly all of the
> > cases I was expecting to encounter.
> 
> Broadly speaking I think there are 3-4 issues to resolve as part of
> moving forward with PEP 649:
> 
> 1) Class decorators (the most relevant being @dataclass) that need to
> inspect something about annotations, and because they run right after
> class definition, the laziness of PEP 649 is not sufficient to allow
> forward references to work. Roughly in a similar boat are `if
> TYPE_CHECKING` use cases where annotations reference names that
> aren't
> ever imported.
> 
> 2) "Documentation" use cases (e.g. built-in "help()") that really
> prefer access to the original text of the annotation, not the repr()
> of the fully-evaluated object -- this is especially relevant if the
> annotation text is a nice short meaningful type alias name, and the
> actual value is some massive unreadable Union type.
> 
> 3) Ensuring that we don't regress import performance too much.
> 
> 4) A solid migration path from the status quo (where many people have
> already started adopting PEP 563) to the best future end state.
> Particularly for libraries that want to support the full range of
> supported Python versions.
> 
> Issues (1) and (2) can be resolved under PEP 649 by providing a way
> to
> run the __co_annotations__ function without erroring on
> not-yet-defined names, I think we have agreement on a plan there.
> Performance of the latest PEP 649 reference implementation does not
> look too bad relative to PEP 563 in my experiments, so I think this
> is
> not an issue -- there are ideas for how we could reduce the overhead
> even further. The migration path is maybe the most difficult issue --
> specifically how to weigh "medium-term migration pain" (which under
> some proposals might last for years) vs "best long-term end state."
> Still working on reaching consensus there, but we have options to
> choose from. Expect a more thorough proposal (probably in the form of
> an update to PEP 649?) sometime after PyCon.
> 
> Carl

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/M3ZPMSHFZBG4MQ4X55DPNAPA7J4T2ZXM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Proto-PEP part 4: The wonderful third option

[Carl Meyer via Python-Dev]

Hi Paul,

On Sun, May 1, 2022 at 3:47 PM Paul Bryan  wrote:
>
> Can someone state what's currently unpalatable about 649? It seemed to 
> address the forward-referencing issues, certainly all of the cases I was 
> expecting to encounter.

Broadly speaking I think there are 3-4 issues to resolve as part of
moving forward with PEP 649:

1) Class decorators (the most relevant being @dataclass) that need to
inspect something about annotations, and because they run right after
class definition, the laziness of PEP 649 is not sufficient to allow
forward references to work. Roughly in a similar boat are `if
TYPE_CHECKING` use cases where annotations reference names that aren't
ever imported.

2) "Documentation" use cases (e.g. built-in "help()") that really
prefer access to the original text of the annotation, not the repr()
of the fully-evaluated object -- this is especially relevant if the
annotation text is a nice short meaningful type alias name, and the
actual value is some massive unreadable Union type.

3) Ensuring that we don't regress import performance too much.

4) A solid migration path from the status quo (where many people have
already started adopting PEP 563) to the best future end state.
Particularly for libraries that want to support the full range of
supported Python versions.

Issues (1) and (2) can be resolved under PEP 649 by providing a way to
run the __co_annotations__ function without erroring on
not-yet-defined names, I think we have agreement on a plan there.
Performance of the latest PEP 649 reference implementation does not
look too bad relative to PEP 563 in my experiments, so I think this is
not an issue -- there are ideas for how we could reduce the overhead
even further. The migration path is maybe the most difficult issue --
specifically how to weigh "medium-term migration pain" (which under
some proposals might last for years) vs "best long-term end state."
Still working on reaching consensus there, but we have options to
choose from. Expect a more thorough proposal (probably in the form of
an update to PEP 649?) sometime after PyCon.

Carl
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/M3FB6QHB2IOMEXDGHFRHYQEDR3KGZPHG/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Proto-PEP part 4: The wonderful third option

[Larry Hastings]

On 5/1/22 15:44, Paul Bryan wrote:
Can someone state what's currently unpalatable about 649? It seemed to 
address the forward-referencing issues, certainly all of the cases I 
was expecting to encounter.



Carl's talk was excellent here; it would be lovely if he would chime in 
and reply.  Here is my almost-certainly-faulty recollection of what he said.


 * PEP 649 doesn't work for code bases that deliberately using
   un-evaluatable expressions but still examine them at runtime. Some
   code bases would have a major circular import problem if they had to
   import every module they use in annotations.  By only importing
   those in "if TYPE_CHECKING" blocks, mypy (which inspects each module
   in isolation) can resolve the references, so it works fine at static
   analysis time.  Occasionally they /also/ need to examine the
   annotation at runtime, but this is only a rudimentary check, so a
   string works fine for them.  So 563 works, but 649 throws e.g. a
   NameError.  Carl proposes a mitigation strategy here: run the
   co_annotations code object with a special globals dict set that
   creates fake objects instead of failing on lookups.
 * PEP 649 is a pain point for libraries using annotations for
   documentation purposes.  The annotation as written may be very
   readable, but evaluating it may turn into a very complicated object,
   and the repr() or str() of that object may be a complicated and
   obscure the original intent.  Carl proposes using much the same
   strategy here; also it /might/ work to use ast.unparse to pull the
   original expression out of the source code, though this seems like
   it would be less reliable.

That's everything I remember... but I was operating on two hours' sleep 
that day.



You might also consult Brett's thread about finding edge cases in PEPs 
484, 563, and 649:


   
https://discuss.python.org/t/finding-edge-cases-for-peps-484-563-and-649-type-annotations/14314/18


Cheers,


//arry/
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YGX4URVSXRNGPFROCHEQ4Q5HRYVIZ22S/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Proto-PEP part 4: The wonderful third option

[Paul Bryan]

Can someone state what's currently unpalatable about 649? It seemed to
address the forward-referencing issues, certainly all of the cases I
was expecting to encounter.

On Sun, 2022-05-01 at 15:35 -0600, Larry Hastings wrote:
> 
> FWIW, I'm in agreement.  My "forward class" proposal(s) were me
> trying to shine a light to find a way forward; I'm in no way adamant
> that we go that direction.  If we can make 649 palatable without
> introducing forward declarations for classes, that's great!  If in
> the future we discover more edge cases that Carl's approach doesn't
> easily solve, we could always revisit it later.  For now it goes in
> the freezer of "ideas we aren't moving forward with".
> 
> /arry
> On 4/29/22 19:08, Guido van Rossum wrote:
>  
> > 
> >  FWIW, Carl presented a talk about his proposed way forward using
> > PEP 649 with some small enhancements to handle cases like
> > dataclasses (*), and it was well received by those present. I
> > personally hope that this means the end of the "forward class
> > declarations" proposals (no matter how wonderful), but the final
> > word is up to the SC.
> > 
> > (*) Mostly fixing the edge cases of the "eval __code__ with tweaked
> > globals" hack that Carl came up with previously, see
> > https://github.com/larryhastings/co_annotations/issues/2#issuecomment-1092432875
> > .
> > 
> > -- 
> > --Guido van Rossum (python.org/~guido)
> > Pronouns: he/him (why is my pronoun here?)
> > 
> > 
> > ___
> > Python-Dev mailing list -- python-dev@python.org
> > To unsubscribe send an email to python-dev-le...@python.org
> > https://mail.python.org/mailman3/lists/python-dev.python.org/
> > Message archived at 
> > https://mail.python.org/archives/list/python-dev@python.org/message/EBDKGKPMOHM674PMUXCVZDRUD5NTIKZB/
> > Code of Conduct: http://python.org/psf/codeofconduct/
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/RXZWTIRVRVHOOKZ6FHIRLQRQBDNCL62X/
> Code of Conduct: http://python.org/psf/codeofconduct/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OIHZVLRYJAT3JHX37U4QPDKJTXTGO3EM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Proto-PEP part 4: The wonderful third option

[Larry Hastings]

FWIW, I'm in agreement.  My "forward class" proposal(s) were me trying 
to shine a light to find a way forward; I'm in no way adamant that we go 
that direction.  If we can make 649 palatable without introducing 
forward declarations for classes, that's great!  If in the future we 
discover more edge cases that Carl's approach doesn't easily solve, we 
could always revisit it later. For now it goes in the freezer of "ideas 
we aren't moving forward with".



//arry/

On 4/29/22 19:08, Guido van Rossum wrote:
 FWIW, Carl presented a talk about his proposed way forward using PEP 
649 with some small enhancements to handle cases like dataclasses (*), 
and it was well received by those present. I personally hope that this 
means the end of the "forward class declarations" proposals (no matter 
how wonderful), but the final word is up to the SC.


(*) Mostly fixing the edge cases of the "eval __code__ with tweaked 
globals" hack that Carl came up with previously, see 
https://github.com/larryhastings/co_annotations/issues/2#issuecomment-1092432875.


--
--Guido van Rossum (python.org/~guido )
/Pronouns: he/him //(why is my pronoun here?)/ 



___
Python-Dev mailing list --python-dev@python.org
To unsubscribe send an email topython-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived 
athttps://mail.python.org/archives/list/python-dev@python.org/message/EBDKGKPMOHM674PMUXCVZDRUD5NTIKZB/
Code of Conduct:http://python.org/psf/codeofconduct/___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/RXZWTIRVRVHOOKZ6FHIRLQRQBDNCL62X/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: __dunder__ = None

[Guido van Rossum]

Non should behave as closely as possible to it not being defined at all. So
return NotImplemented.

On Sun, May 1, 2022 at 09:53 Patrick Reader <_...@pxeger.com> wrote:

> On 01/05/2022 06:20, Serhiy Storchaka wrote:
> > The question is how to interpret value None:
> >
> > * Always raise TypeError (with changed message)? This is what happen
> > currently when you set the method to None, this is the most compatible
> > option.
> > * Always raise an error, but allow to change it to more appropriate
> > type (for example AttributeError for __setattr__)?
> > * Interpret value None the same way as an absent attribute?
> What about binary operators (__add__, __eq__, etc)? Should they act as
> if they'd returned NotImplemented? Or immediately unconditionally raise
> a TypeError?
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/PLWKIT7FWXLKIGQXL3X5GFT3MGTC53R3/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-- 
--Guido (mobile)
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JNFF5Q6M2AU2YKZP52XTAL6HHDMGFRV7/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: __dunder__ = None

[Patrick Reader]

On 01/05/2022 06:20, Serhiy Storchaka wrote:

The question is how to interpret value None:

* Always raise TypeError (with changed message)? This is what happen 
currently when you set the method to None, this is the most compatible 
option.
* Always raise an error, but allow to change it to more appropriate 
type (for example AttributeError for __setattr__)?

* Interpret value None the same way as an absent attribute?
What about binary operators (__add__, __eq__, etc)? Should they act as 
if they'd returned NotImplemented? Or immediately unconditionally raise 
a TypeError?

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PLWKIT7FWXLKIGQXL3X5GFT3MGTC53R3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 689 – Semi-stable C API tier

[Nick Coghlan]

On Sat, 30 Apr 2022, 3:02 am Guido van Rossum,  wrote:

>
> I think picking "semi-stable" would be giving in to the OCD nerd in all of
> us. :-) While perhaps technically less precise, "unstable" is the catchy
> name with the right association. (And yes, we should keep it stable within
> bugfix releases, but the name doesn't need to reflect that detail.) The
> "internal API" isn't an API at all (except for CPython core developers and
> contributors). The "unstable API" would definitely be an *API* for users
> outside the core.
>
> So let's please go with "unstable".
>

While I've advocated for semi-stable in previous threads, I now agree the
pragmatic arguments for "unstable" hold up well enough to make the simpler
term the better choice:

* no question around using a hyphen or not
* "unstable public C API" is sufficient to distinguish the new tier from
Py_BUILD_CORE's completely unstable internal API

The risks of misinterpretation are also low:

* external users that need one of these APIs will presumably be invested
enough to actually check the stability expectations in the docs
* core devs will have regression tests to remind us that the published
unstable APIs aren't allowed to change after beta 1

Cheers,
Nick.




> --
> --Guido van Rossum (python.org/~guido)
> *Pronouns: he/him **(why is my pronoun here?)*
> 
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6TVQ6MEVKNLYNQAHMQHCHDMJES7RIRXL/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: __dunder__ = None

[Serhiy Storchaka]

01.05.22 13:02, Spencer Brown пише:
It actually is documented as being supported here: 
https://docs.python.org/3.10/reference/datamodel.html#id2 
, and as 
mentioned there __iter__(), __reversed__() and __contains__() also have 
special support so they avoid trying to fallback to __getitem__ etc. 
Also some discussion at https://github.com/python/cpython/issues/70146 
. For most methods the 
conclusion back then makes sense, a specific exception message being 
raised would just potentially slow the interpreter.


Thank you Spencer. It answers my questions.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LNE2COY3OAV7EJOMPBFBHSVPBTYM4KUQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: __dunder__ = None

[Spencer Brown]

It actually is documented as being supported here: 
https://docs.python.org/3.10/reference/datamodel.html#id2, and as mentioned 
there __iter__(), __reversed__() and __contains__() also have special support 
so they avoid trying to fallback to __getitem__ etc. Also some discussion at 
https://github.com/python/cpython/issues/70146. For most methods the conclusion 
back then makes sense, a specific exception message being raised would just 
potentially slow the interpreter.

- Spencer

On 1 May 2022, at 3:22 pm, Serhiy Storchaka  wrote:

There is a special handling of `__hash__` set to None in the interpreter core. 
This is because every class inherited the `__hash__` attribute from "object", 
and setting `__hash__ = None` is a simple way to make it unhashable. It makes 
hash() raising the correct type of exception (TypeError), but with unhelpful 
error message "'NoneType' object is not callable". The special case was added 
to make the error message more relevant: "unhashable type: '{typename}'".

There is similar situation with other special methods defined in "object" or 
other common classes. Sometimes we want to cancel the default inherited 
behavior.

  >>> dir(object)
  ['__class__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', 
'__ge__', '__getattribute__', '__getstate__', '__gt__', '__hash__', '__init__', 
'__init_subclass__', '__le__', '__lt__', '__ne__', '__new__', '__reduce__', 
'__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', 
'__subclasshook__']

I propose to support officially the idiom "__dunder__ = None" and add special 
cases to raise more specialized exception instead of "TypeError: 'NoneType' 
object is not callable" for most of special method where cancelling the default 
behavior makes sense (for example I do not think that we need better error 
message for `__repr__ = None`).

The question is how to interpret value None:

* Always raise TypeError (with changed message)? This is what happen currently 
when you set the method to None, this is the most compatible option.
* Always raise an error, but allow to change it to more appropriate type (for 
example AttributeError for __setattr__)?
* Interpret value None the same way as an absent attribute?

For `__hash__` or `__class_getitem__` all three options mean the same. But 
absent `__mro_entries__` and `__mro_entries__ = None` currently give different 
results. It is even more complicated for pickling: absent `__reduce_ex__` and 
`__reduce_ex__ = None` mean the same in the Python implementation, but give 
different results in the C implementation.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YGAK34DRWJFSIV2VZ4NC2J24XO37GCMM/
Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VFNIUX6K6ZVLUOHNDSD7NGPCWGLESLE7/
Code of Conduct: http://python.org/psf/codeofconduct/