Hi all,
I got pinged to voice my opinion on PEP 649 as the instigator of PEP 563. I'm
sorry, this is long, and a separate thread, because it deals with three things:
- Goals set for PEP 563 and how it did in practice;
- PEP 649 and how it addresses those same goals;
- can we cleanly adopt PEP 649?
First off, it looks like this isn't worded clearly enough in the PEP itself so
let me summarize what the goals of PEP 563 were:
Goal 1. To get rid of the forward reference problem, e.g. when a type is
declared lower in the file than its use. A cute special case of this is when a
class has a method that accepts or returns objects of its own type.
Goal 2. To somewhat decouple the syntax of type annotations from the runtime
requirements, allowing for better expressibility.
Goal 3. To make annotations affect runtime characteristics of typed modules
less, namely import time and memory usage.
Now, did PEP 563 succeed in its goals? Well, partially at best. Let's see.
In terms of Goal 1, it turned out that `typing.get_type_hints()` has limits
that make its use in general costly at runtime, and more importantly
insufficient to resolve all types. The most common example deals with
non-global context in which types are generated (e.g. inner classes, classes
within functions, etc.). But one of the crown examples of forward references:
classes with methods accepting or returning objects of their own type, also
isn't properly handled by `typing.get_type_hints()` if a class generator is
used. There's some trickery we can do to connect the dots but in general it's
not great.
As for Goal 2, it became painfully obvious that a number of types used for
static typing purposes live outside of the type annotation context. So while
PEP 563 tried to enable a backdoor for more usable static typing syntax, it
ultimately couldn't. This is where PEP 585 and later PEP 604 came in, filling
the gap by doing the sad but necessary work of enabling this extended typing
syntax in proper runtime Python context. This is what should have been done all
along and it makes PEP 563 in this context irrelevant as of Python 3.9 (for PEP
585) and 3.10 (for PEP 604). However, to the extent of types used within
annotations, the PEP 563 future-import allows using the new cute typing syntax
already for Python 3.7+ compatible code. So library authors can already adopt
the lowercase type syntax of PEP 585 and the handy pipe syntax for unions of
PEP 604. And even for non-type annotation uses that can be successfully barred
by a `if TYPE_CHECKING` block, like type aliases, type variables, and such. Of
course that has no chance of working with `typing.get_type_hints()`.
Now, Goal 3 is a different matter. As Inada Naoki demonstrated somewhere in the
preceding discussion here, PEP 563 made fully type-annotationed codebase import
pretty much as fast as non-annotated code, and through the joys of string
interning, use relatively little extra memory. At the time PEP 563, a popular
concern around static typing in Python was that it slows down runtime while
it's only useful for static analysis. While we (the typing crowd) were always
sure the "only useful as a better linter" is dismissive, the performance
argument had to go.
So where does this leave us today?
Runtime use of types was somewhat overly optimistically treated as solvable
with `typing.get_type_hints()`. Now Pydantic and other similar tools show that
this isn't sadly the case. Without the future-import, they could ignore the
problem until Python 3.10 but no longer. I was somewhat surprised this was the
case because forward references as strings could always be used. So I guess the
answer there was to just not use them if you want your runtime tool to work.
Fair enough.
PEP 649 addresses this runtime usage of type annotations head on in a way that
eluded me when I first set out to solve this problem. Back then, Larry and Mark
Shannon did voice their opinion that through some clever frame object storage,
"implicit lambdas", we can address the issue of forward references. This seemed
unattractive to me at the time because it didn't deal with our Goal 2 and our
understanding was that it actually makes Goal 3 worse by holding on to all
frames in memory where type annotations appear, and by creating massive
duplication of equivalent annotations in memory due to lack of a mechanism
similar to string interning. Now those issues are somewhat solved in the final
PEP 649 and this makes for an interesting compromise for us to make. I say
"compromise" because as Inada Naoki measured, there's still a non-zero
performance cost of PEP 649 versus PEP 563:
- code size: +63%
- memory: +62%
- import time: +60%
Will this hurt some current users of typing? Yes, I can name you multiple past
employers of mine where this will be the case. Is it worth it for Pydantic? I
tend to think that yes, it is, since it is a significant community, and the
operations on type annotations it performs are in the sensible set for which
`typing.get_type_hints()` was proposed.
However, there are some big open questions about how to adopt PEP 649.
Question 1. What should happen to code that already adopted `from __future__
import annotations`? Future imports were never thought of as feature toggles so
if PEP 563 isn't going to become the default, it should get removed. However,
removing it needs a deprecation period, otherwise code that already adopted the
future-import will fail to execute (even if you leave a dummy future-import
there -- forward references, remember?). So deprecation, and with that, a
rather tricky situation where PEP 649 will have to support files with the
future-import. Now PEP 649 says that an object cannot both have __annotations__
and __co_annotations__ set at the same time though, so I guess files with the
future-import would necessarily be treated as some deprecated code that might
or might not be translatable to PEP 649 co-annotations.
Question 2. If PEP 649 deprecates PEP 563, will the use of the future-import
for early adoption of PEP 585 and PEP 604 syntax become disallowed? Ultimately
this is my biggest worry -- that there doesn't seem to be a clear adoption path
for this change. If we just take it wholesale and declare PEP 563 a failure,
that bars library authors from using PEP 585/604 syntax until Python 3.10
becomes their lowest supported version. That's October 2025 at the earliest if
we're looking at 3.9 lifespan. That would be a significant wrench thrown in
typing adoption as PEP 585 and 604 provide significant usability advantages, to
the point where often modules with non-trivial annotations don't even have to
import the typing module at all.
Question 3. Additionally, it's unclear to me whether PEP 649 allows for any
relaxed syntax in type annotations that might not be immediately valid in a
given version of Python. Say, hypothetically (please don't bikeshed this!), we
adopt PEP 649 in Python 3.10 and in Python 3.11 we come up with a shorthand for
optional types using a question mark. Can I put the question mark there in
Python 3.10 at all? Second made up example: let's say in Python 3.12 we allow
passing a function as a type (meaning "a callable just LIKE this one"). Can I
put this callable there now in Python 3.10?
Now, should we postpone with until Python 3.11? We should not, at least not in
terms of figuring out a clear migration path from here to there, ideally such
that existing end users of the PEP 563 future-import can just live their lives
as if nothing ever happened. Given that the goal of the future-import was
largely to say "I don't care about those annotations at runtime", maybe PEP 649
could somehow adopt the existing future-import? `typing.get_type_hints()` would
do what it does today anyway. The only users affected would be those directly
using the strings in PEP 563 annotations, and it's unclear whether this is a
real problem. There are a hundred pages of results on GitHub [1]_ that include
`__annotations__` so this would have to be sought through.
If we don't make it in time for including this in Python 3.10, then so be it.
But let's not wait until April 2022 ;-)
i-regret-nothing'ly yours,
Łukasz
_[1] https://github.com/search?l=Python&q=__annotations__&type=Code
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/ZBJ7MD6CSGM6LZAOTET7GXAVBZB7O77O/
Code of Conduct: http://python.org/psf/codeofconduct/
