[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2
On 4/11/21 7:55 PM, Paul Bryan wrote: PEP 563 also requires using ``eval()`` or ``typing.get_type_hints()`` to examine annotations. Code updated to work with PEP 563 that calls ``eval()`` directly would have to be updated simply to remove the ``eval()`` call. Code using ``typing.get_type_hints()`` would continue to work unchanged, though future use of that function would become optional in most cases. I think it is worth noting somewhere that string annotations are still valid, and should still be evaluated if so. That's not up to me, it's up to the static type checkers who created that idiom. But I assume they'll continue to support stringized annotations, whether manually or automatically created. Because this PEP makes semantic changes to how annotations are evaluated, this PEP will be initially gated with a per-module ``from __future__ import co_annotations`` before it eventually becomes the default behavior. Is it safe to assume that a module that does not import co_annotations, but imports a module that does, will exhibit PEP 649 behavior when the former accesses an annotation defined in the latter? Yes. * *Code that sets annotations on module or class attributes from inside any kind of flow control statement.* It's currently possible to set module and class attributes with annotations inside an ``if`` or ``try`` statement, and it works as one would expect. It's untenable to support this behavior when this PEP is active. Is the following an example of the above? @dataclass class Foo: if some_condition: x: int else: x: float If so, would the following still be valid? if some_condition: type_ = int else: type_ = float @dataclass class Foo: x: type_ Your example was valid, and I think your workaround should be fine. Do you have a use case for this, or is this question motivated purely by curiosity? * *Code in module or class scope that references or modifies the local* ``__annotations__`` *dict directly.* Currently, when setting annotations on module or class attributes, the generated code simply creates a local ``__annotations__`` dict, then sets mappings in it as needed. It's also possible for user code to directly modify this dict, though this doesn't seem like it's an intentional feature. Although it would be possible to support this after a fashion when this PEP was active, the semantics would likely be surprising and wouldn't make anyone happy. I recognize the point you make later about its impact on static type checkers. Setting that aside, I'm wondering about caes where annotations can be dynamically generated, such as dataclasses.make_dataclass(...). And, I could see reasons for overwriting values in __annotations__, especially in the case where it may be stored as a string and one wants to later affix its evaluated value. These are considerations specific to runtime (dynamic) type checking. It's fine to modify the __annotations__ dict after the creation of the class or module. It's code that modifies "__annotations__" from within the class or module that is disallowed here. Similarly for dataclasses; once it creates a class object, it can explicitly set and / or modify the annotations dict on that class. I wonder if it would make sense for each item in __annotations__ to be evaluated separately on first access /of each key/, rather than all __annotations__ on first access to the dict. Basically the dict would act as a LazyDict. It could also provide the benefit of lessening the expense of evaluating complex but otherwise unused annotations. This would cause an immense proliferation of code objects (with some pre-bound to function objects). Rather than one code object per annotation dict, it would create one code object per annotation key. Also, we don't have a "lazy dict" object built in to Python, so we'd have to create one. I don't have any problems that this would solve, so I'm not super interested in it. Personally I'd want to see a real compelling use case for this feature before I'd consider adding it to Python. Of course, I'm not on the steering committee, so my opinion is only worth so much. //arry/ ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/WTPNW5YWODQGR66GMB2OJYYMUQDDSPHZ/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Relaxing the annotation syntax
I'm in favour of the approach proposed in PEP 649.
Movie trailer: "In a world where annotations are arbitrary non-Python
syntax..."
It seems to me we could always have annotations evaluate to Python
expressions *and* support any arbitrary syntax (e.g. through
Annotated[...] or similar mechanism). What would a relaxed inline
syntax provide that a well-placed Annotated[type,
ArbitraryNonPythonSyntax("...")] annotation wouldn't? .
Paul
On Sun, 2021-04-11 at 20:43 -0700, Guido van Rossum wrote:
> On Sun, Apr 11, 2021 at 1:31 PM Barry Warsaw
> wrote:
> [snip]
> > This is something the SC has been musing about, but as it’s not a
> > fully formed idea, I’m a little hesitant to bring it up. That
> > said, it’s somewhat relevant: We wonder if it may be time to in a
> > sense separate the typing syntax from Python’s regular syntax.
> > TypeGuards are a case where if typing had more flexibility to adopt
> > syntax that wasn’t strictly legal “normal” Python, maybe something
> > more intuitive could have been proposed. I wonder if the typing-
> > sig has discussed this possibility (in the future, of course)?
> >
>
>
> We haven't discussed this in typing-sig, but it so happens that a
> similar idea for JavaScript was mentioned to me recently, and at the
> time I spent about 5 seconds thinking about how this could be useful
> for Python, too.
>
> Basically, where the original PEP 3107 proposed annotations to have
> the syntax of expressions and evaluate them as such, now that we've
> got PEP 563 which makes annotations available as strings and no
> longer attempts to evaluate them, we could relax this further and do
> something like just skipping tokens until a suitable delimiter is
> found (',' or ')' inside the parameter list, ':' for the return
> type). Of course, matching parentheses, brackets and braces should
> always be paired and the target delimiter should not terminate the
> scan inside such matched pairs.
>
> It occurs to me that right now is actually very good time to think
> about this a little more, because we're at a crossroads, of sorts: we
> could adopt Larry Hastings' PEP 649, which reverses PEP 563 and makes
> annotations available at runtime as objects (e.g., `def f(x: int)`
> would have the `int` type object in the annotation instead of the
> string `"int"`). Or we could reject PEP 649, which leaves the door
> open for a more relaxed annotation syntax in the future (earliest in
> 3.11).
>
> At the very least I recommend that the SC take this into account when
> they consider PEP 649. Accepting it has some nice benefits when it
> comes to the scoping rules for annotations -- but it would forever
> close the door for the "relaxed annotation syntax" idea you brought
> up. (Isn't it fun to be on the SC. :-)
>
> [snip]
> > Agreed. It’s interesting that PEP 593 proposes a different
> > approach to enriching the typing system. Typing itself is becoming
> > a little ecosystem of its own, and given that many Python users are
> > still not fully embracing typing, maybe continuing to tie the
> > typing syntax to Python syntax is starting to strain.
>
> It definitely is. Type checkers are still young compared to Python
> itself, and their development speed is much faster than that of
> Python. So whenever new syntax is required the strain becomes
> obvious. Thanks for making that observation!
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/2F5PVC5MOWMGFVOX6FUQOUC7EJEEXFN3/
> Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/PUO7U3ZUIJ24W32GM7PYT34RJ5MGWZOC/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Relaxing the annotation syntax
On Sun, Apr 11, 2021 at 1:31 PM Barry Warsaw wrote:
[snip]
> This is something the SC has been musing about, but as it’s not a fully
> formed idea, I’m a little hesitant to bring it up. That said, it’s
> somewhat relevant: We wonder if it may be time to in a sense separate the
> typing syntax from Python’s regular syntax. TypeGuards are a case where if
> typing had more flexibility to adopt syntax that wasn’t strictly legal
> “normal” Python, maybe something more intuitive could have been proposed.
> I wonder if the typing-sig has discussed this possibility (in the future,
> of course)?
>
We haven't discussed this in typing-sig, but it so happens that a similar
idea for JavaScript was mentioned to me recently, and at the time I spent
about 5 seconds thinking about how this could be useful for Python, too.
Basically, where the original PEP 3107 proposed annotations to have the
syntax of expressions and evaluate them as such, now that we've got PEP 563
which makes annotations available as strings and no longer attempts to
evaluate them, we could relax this further and do something like just
skipping tokens until a suitable delimiter is found (',' or ')' inside the
parameter list, ':' for the return type). Of course, matching parentheses,
brackets and braces should always be paired and the target delimiter should
not terminate the scan inside such matched pairs.
It occurs to me that right now is actually very good time to think about
this a little more, because we're at a crossroads, of sorts: we could adopt
Larry Hastings' PEP 649, which reverses PEP 563 and makes annotations
available at runtime as objects (e.g., `def f(x: int)` would have the `int`
type object in the annotation instead of the string `"int"`). Or we could
reject PEP 649, which leaves the door open for a more relaxed annotation
syntax in the future (earliest in 3.11).
At the very least I recommend that the SC take this into account when they
consider PEP 649. Accepting it has some nice benefits when it comes to the
scoping rules for annotations -- but it would forever close the door for
the "relaxed annotation syntax" idea you brought up. (Isn't it fun to be on
the SC. :-)
[snip]
> Agreed. It’s interesting that PEP 593 proposes a different approach to
> enriching the typing system. Typing itself is becoming a little ecosystem
> of its own, and given that many Python users are still not fully embracing
> typing, maybe continuing to tie the typing syntax to Python syntax is
> starting to strain.
>
It definitely is. Type checkers are still young compared to Python itself,
and their development speed is much faster than that of Python. So whenever
new syntax is required the strain becomes obvious. Thanks for making that
observation!
--
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*
[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2
I like! I really appreciate the work you've put into this to get it this far. Questions and comments: > PEP 563 also requires using ``eval()`` or ``typing.get_type_hints()`` > to examine annotations. Code updated to work with PEP 563 that calls > ``eval()`` directly would have to be updated simply to remove the > ``eval()`` call. Code using ``typing.get_type_hints()`` would > continue to work unchanged, though future use of that function > would become optional in most cases. I think it is worth noting somewhere that string annotations are still valid, and should still be evaluated if so. > Because this PEP makes semantic changes to how annotations are > evaluated, this PEP will be initially gated with a per-module > ``from __future__ import co_annotations`` before it eventually > becomes the default behavior. Is it safe to assume that a module that does not import co_annotations, but imports a module that does, will exhibit PEP 649 behavior when the former accesses an annotation defined in the latter? > * *Code that sets annotations on module or class attributes > from inside any kind of flow control statement.* It's > currently possible to set module and class attributes with > annotations inside an ``if`` or ``try`` statement, and it works > as one would expect. It's untenable to support this behavior > when this PEP is active. Is the following an example of the above? @dataclass class Foo: if some_condition: x: int else: x: float If so, would the following still be valid? if some_condition: type_ = int else: type_ = float @dataclass class Foo: x: type_ > * *Code in module or class scope that references or modifies the > local* ``__annotations__`` *dict directly.* Currently, when > setting annotations on module or class attributes, the generated > code simply creates a local ``__annotations__`` dict, then sets > mappings in it as needed. It's also possible for user code > to directly modify this dict, though this doesn't seem like it's > an intentional feature. Although it would be possible to support > this after a fashion when this PEP was active, the semantics > would likely be surprising and wouldn't make anyone happy. I recognize the point you make later about its impact on static type checkers. Setting that aside, I'm wondering about caes where annotations can be dynamically generated, such as dataclasses.make_dataclass(...). And, I could see reasons for overwriting values in __annotations__, especially in the case where it may be stored as a string and one wants to later affix its evaluated value. These are considerations specific to runtime (dynamic) type checking. I wonder if it would make sense for each item in __annotations__ to be evaluated separately on first access of each key, rather than all __annotations__ on first access to the dict. Basically the dict would act as a LazyDict. It could also provide the benefit of lessening the expense of evaluating complex but otherwise unused annotations. Paul On Sun, 2021-04-11 at 18:55 -0700, Larry Hastings wrote: > > Attached is my second draft of PEP 649. The PEP and the prototype > have both seen a marked improvement since round 1 in January; PEP 649 > now allows annotations to refer to any variable they could see under > stock semantics: > * Local variables in the current function scope or in enclosing >function scopes become closures and use LOAD_DEFER. > * Class variables in the current class scope are made available using >a new mechanism, in which the class dict is attached to the bound >annotation function, then loaded into f_locals when the annotation >function is run. Thus permitting LOAD_NAME opcodes to function >normally. > > I look forward to your comments, > > /arry > ___ > Python-Dev mailing list -- python-dev@python.org > To unsubscribe send an email to python-dev-le...@python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/QSASX6PZ3LIIFIANHQQFS752BJYFUFPY/ > Code of Conduct: http://python.org/psf/codeofconduct/ ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/SUSRZHLME3QQ4THICJVMQKRH4RVRY6XL/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2
Attached is my second draft of PEP 649. The PEP and the prototype have
both seen a marked improvement since round 1 in January; PEP 649 now
allows annotations to refer to any variable they could see under stock
semantics:
* Local variables in the current function scope or in enclosing
function scopes become closures and use LOAD_DEFER.
* Class variables in the current class scope are made available using
a new mechanism, in which the class dict is attached to the bound
annotation function, then loaded into f_locals when the annotation
function is run. Thus permitting LOAD_NAME opcodes to function
normally.
I look forward to your comments,
//arry/
PEP: 649
Title: Deferred Evaluation Of Annotations Using Descriptors
Author: Larry Hastings
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 11-Jan-2021
Post-History: 11-Jan-2021, 11-Apr-2021
Abstract
As of Python 3.9, Python supports two different behaviors
for annotations:
* original or "stock" Python semantics, in which annotations
are evaluated at the time they are bound, and
* PEP 563 semantics, currently enabled per-module by
``from __future__ import annotations``, in which annotations
are converted back into strings and must be reparsed and
executed by ``eval()`` to be used.
Original Python semantics created a circular references problem
for static typing analysis. PEP 563 solved that problem--but
its novel semantics introduced new problems, including its
restriction that annotations can only reference names at
module-level scope.
This PEP proposes a third way that embodies the best of both
previous approaches. It solves the same circular reference
problems solved by PEP 563, while otherwise preserving Python's
original annotation semantics, including allowing annotations
to refer to local and class variables.
In this new approach, the code to generate the annotations
dict is written to its own function which computes and returns
the annotations dict. Then, ``__annotations__`` is a "data
descriptor" which calls this annotation function once and
retains the result. This delays the evaluation of annotations
expressions until the annotations are examined, at which point
all circular references have likely been resolved. And if
the annotations are never examined, the function is never
called and the annotations are never computed.
Annotations defined using this PEP's semantics have the same
visibility into the symbol table as annotations under "stock"
semantics--any name visible to an annotation in Python 3.9
is visible to an annotation under this PEP. In addition,
annotations under this PEP can refer to names defined *after*
the annotation is defined, as long as the name is defined in
a scope visible to the annotation. Specifically, when this PEP
is active:
* An annotation can refer to a local variable defined in the
current function scope.
* An annotation can refer to a local variable defined in an
enclosing function scope.
* An annotation can refer to a class variable defined in the
current class scope.
* An annotation can refer to a global variable.
And in all four of these cases, the variable referenced by
the annotation needn't be defined at the time the annotation
is defined--it can be defined afterwards. The only restriction
is that the name or variable be defined before the annotation
is *evaluated.*
If accepted, these new semantics for annotations would initially
be gated behind ``from __future__ import co_annotations``.
However, these semantics would eventually be promoted to be
Python's default behavior. Thus this PEP would *supersede*
PEP 563, and PEP 563's behavior would be deprecated and
eventually removed.
Overview
.. note:: The code presented in this section is simplified
for clarity. The intention is to communicate the high-level
concepts involved without getting lost in with the details.
The actual details are often quite different. See the
Implementation_ section later in this PEP for a much more
accurate description of how this PEP works.
Consider this example code::
def foo(x: int = 3, y: MyType = None) -> float:
...
class MyType:
...
foo_y_type = foo.__annotations__['y']
As we see here, annotations are available at runtime through an
``__annotations__`` attribute on functions, classes, and modules.
When annotations are specified on one of these objects,
``__annotations__`` is a dictionary mapping the names of the
fields to the value specified as that field's annotation.
The default behavior in Python 3.9 is to evaluate the expressions
for the annotations, and build the annotations dict, at the time
the function, class, or module is bound. At runtime the above
code actually works something like this::
annotations = {'x': int, 'y': MyType, 'return': float}
def foo(x = 3, y = "abc"):
...
foo.__annotations__ = annotations
class MyType:
...
foo_y_type =
[Python-Dev] Re: PEP 647 Accepted
On Apr 7, 2021, at 12:59, Guido van Rossum wrote:
>
> Note that in TypeScript this also doesn't look like a boolean -- it uses a
> unique syntax that has to be learned:
>
> function isCustomer(partner: any): partner is Customer {
> . . .
> }
>
> Arguably the TS syntax is more easily intuited without looking it up, but TS
> has a certain freedom in its syntactic design that we don't have for Python:
> new *syntax* has to be added to the Python parser and can't be backported,
> whereas new *types* (like `TypeGuard[T]`) can easily be backported via
> typing_extensions.py.
Thanks Guido. Yes, we totally understand. I agree that this TS example is
easier to reason about (at least for me), and that Python is limited in what
syntax it can allow there.
This is something the SC has been musing about, but as it’s not a fully formed
idea, I’m a little hesitant to bring it up. That said, it’s somewhat relevant:
We wonder if it may be time to in a sense separate the typing syntax from
Python’s regular syntax. TypeGuards are a case where if typing had more
flexibility to adopt syntax that wasn’t strictly legal “normal” Python, maybe
something more intuitive could have been proposed. I wonder if the typing-sig
has discussed this possibility (in the future, of course)?
> We have really tried, but we did not come up with anything better than the
> current PEP.
Neither did the SC, thus the acceptance! :D I have no doubt typing-sig really
tried hard!
> FWIW you might be interested in Annotated (PEP 593), which can be used to
> indicate various attributes of a type annotation. Before you suggest that we
> adopt that instead of PEP 647, we considered that, and the consensus is that
> that's not what Annotated is for (it's intended for conveying information to
> tools *other* than the type checker, for example schema checkers etc.).
Agreed. It’s interesting that PEP 593 proposes a different approach to
enriching the typing system. Typing itself is becoming a little ecosystem of
its own, and given that many Python users are still not fully embracing typing,
maybe continuing to tie the typing syntax to Python syntax is starting to
strain.
Cheers,
-Barry
signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/YMNPYKWVM537V7A6LJXG5R3GUCOTBYTL/
Code of Conduct: http://python.org/psf/codeofconduct/
