Łukasz, I am hereby accepting your PEP. This will be a great improvement in the experience of users annotating large complex codebases. Congrats on the design and implementation and on your shepherding the PEP through the discussion phase. Also a special thanks to Serhiy for thoroughly reviewing and contributing to the ast-expr-stringification code.
--Guido PS. I have some editorial quibbles (mostly suggestions to make the exposition clearer in a few places) but they don't affect acceptance of the PEP and I will contact you at a later time with these. On Tue, Nov 21, 2017 at 4:26 PM, Lukasz Langa <luk...@langa.pl> wrote: > Based on the feedback I gather in early November, > I'm publishing the third draft for consideration on python-dev. > I hope you like it! > > A nicely formatted rendering is available here: > https://www.python.org/dev/peps/pep-0563/ > > The full list of changes between this version and the previous draft > can be found here: > https://github.com/ambv/static-annotations/compare/ > python-dev1...python-dev2 > > - Ł > > > > PEP: 563 > Title: Postponed Evaluation of Annotations > Version: $Revision$ > Last-Modified: $Date$ > Author: Łukasz Langa <luk...@langa.pl> > Discussions-To: Python-Dev <python-dev@python.org> > Status: Draft > Type: Standards Track > Content-Type: text/x-rst > Created: 8-Sep-2017 > Python-Version: 3.7 > Post-History: 1-Nov-2017, 21-Nov-2017 > Resolution: > > > Abstract > ======== > > PEP 3107 introduced syntax for function annotations, but the semantics > were deliberately left undefined. PEP 484 introduced a standard meaning > to annotations: type hints. PEP 526 defined variable annotations, > explicitly tying them with the type hinting use case. > > This PEP proposes changing function annotations and variable annotations > so that they are no longer evaluated at function definition time. > Instead, they are preserved in ``__annotations__`` in string form. > > This change is going to be introduced gradually, starting with a new > ``__future__`` import in Python 3.7. > > > Rationale and Goals > =================== > > PEP 3107 added support for arbitrary annotations on parts of a function > definition. Just like default values, annotations are evaluated at > function definition time. This creates a number of issues for the type > hinting use case: > > * forward references: when a type hint contains names that have not been > defined yet, that definition needs to be expressed as a string > literal; > > * type hints are executed at module import time, which is not > computationally free. > > Postponing the evaluation of annotations solves both problems. > > Non-goals > --------- > > Just like in PEP 484 and PEP 526, it should be emphasized that **Python > will remain a dynamically typed language, and the authors have no desire > to ever make type hints mandatory, even by convention.** > > This PEP is meant to solve the problem of forward references in type > annotations. There are still cases outside of annotations where > forward references will require usage of string literals. Those are > listed in a later section of this document. > > Annotations without forced evaluation enable opportunities to improve > the syntax of type hints. This idea will require its own separate PEP > and is not discussed further in this document. > > Non-typing usage of annotations > ------------------------------- > > While annotations are still available for arbitrary use besides type > checking, it is worth mentioning that the design of this PEP, as well > as its precursors (PEP 484 and PEP 526), is predominantly motivated by > the type hinting use case. > > In Python 3.8 PEP 484 will graduate from provisional status. Other > enhancements to the Python programming language like PEP 544, PEP 557, > or PEP 560, are already being built on this basis as they depend on > type annotations and the ``typing`` module as defined by PEP 484. > In fact, the reason PEP 484 is staying provisional in Python 3.7 is to > enable rapid evolution for another release cycle that some of the > aforementioned enhancements require. > > With this in mind, uses for annotations incompatible with the > aforementioned PEPs should be considered deprecated. > > > Implementation > ============== > > In Python 4.0, function and variable annotations will no longer be > evaluated at definition time. Instead, a string form will be preserved > in the respective ``__annotations__`` dictionary. Static type checkers > will see no difference in behavior, whereas tools using annotations at > runtime will have to perform postponed evaluation. > > The string form is obtained from the AST during the compilation step, > which means that the string form might not preserve the exact formatting > of the source. Note: if an annotation was a string literal already, it > will still be wrapped in a string. > > Annotations need to be syntactically valid Python expressions, also when > passed as literal strings (i.e. ``compile(literal, '', 'eval')``). > Annotations can only use names present in the module scope as postponed > evaluation using local names is not reliable (with the sole exception of > class-level names resolved by ``typing.get_type_hints()``). > > Note that as per PEP 526, local variable annotations are not evaluated > at all since they are not accessible outside of the function's closure. > > Enabling the future behavior in Python 3.7 > ------------------------------------------ > > The functionality described above can be enabled starting from Python > 3.7 using the following special import:: > > from __future__ import annotations > > A reference implementation of this functionality is available > `on GitHub <https://github.com/ambv/cpython/tree/string_annotations>`_. > > > Resolving Type Hints at Runtime > =============================== > > To resolve an annotation at runtime from its string form to the result > of the enclosed expression, user code needs to evaluate the string. > > For code that uses type hints, the > ``typing.get_type_hints(obj, globalns=None, localns=None)`` function > correctly evaluates expressions back from its string form. Note that > all valid code currently using ``__annotations__`` should already be > doing that since a type annotation can be expressed as a string literal. > > For code which uses annotations for other purposes, a regular > ``eval(ann, globals, locals)`` call is enough to resolve the > annotation. > > In both cases it's important to consider how globals and locals affect > the postponed evaluation. An annotation is no longer evaluated at the > time of definition and, more importantly, *in the same scope* where it > was defined. Consequently, using local state in annotations is no > longer possible in general. As for globals, the module where the > annotation was defined is the correct context for postponed evaluation. > > The ``get_type_hints()`` function automatically resolves the correct > value of ``globalns`` for functions and classes. It also automatically > provides the correct ``localns`` for classes. > > When running ``eval()``, > the value of globals can be gathered in the following way: > > * function objects hold a reference to their respective globals in an > attribute called ``__globals__``; > > * classes hold the name of the module they were defined in, this can be > used to retrieve the respective globals:: > > cls_globals = vars(sys.modules[SomeClass.__module__]) > > Note that this needs to be repeated for base classes to evaluate all > ``__annotations__``. > > * modules should use their own ``__dict__``. > > The value of ``localns`` cannot be reliably retrieved for functions > because in all likelihood the stack frame at the time of the call no > longer exists. > > For classes, ``localns`` can be composed by chaining vars of the given > class and its base classes (in the method resolution order). Since slots > can only be filled after the class was defined, we don't need to consult > them for this purpose. > > Runtime annotation resolution and class decorators > -------------------------------------------------- > > Metaclasses and class decorators that need to resolve annotations for > the current class will fail for annotations that use the name of the > current class. Example:: > > def class_decorator(cls): > annotations = get_type_hints(cls) # raises NameError on 'C' > print(f'Annotations for {cls}: {annotations}') > return cls > > @class_decorator > class C: > singleton: 'C' = None > > This was already true before this PEP. The class decorator acts on > the class before it's assigned a name in the current definition scope. > > Runtime annotation resolution and ``TYPE_CHECKING`` > --------------------------------------------------- > > Sometimes there's code that must be seen by a type checker but should > not be executed. For such situations the ``typing`` module defines a > constant, ``TYPE_CHECKING``, that is considered ``True`` during type > checking but ``False`` at runtime. Example:: > > import typing > > if typing.TYPE_CHECKING: > import expensive_mod > > def a_func(arg: expensive_mod.SomeClass) -> None: > a_var: expensive_mod.SomeClass = arg > ... > > This approach is also useful when handling import cycles. > > Trying to resolve annotations of ``a_func`` at runtime using > ``typing.get_type_hints()`` will fail since the name ``expensive_mod`` > is not defined (``TYPE_CHECKING`` variable being ``False`` at runtime). > This was already true before this PEP. > > > Backwards Compatibility > ======================= > > This is a backwards incompatible change. Applications depending on > arbitrary objects to be directly present in annotations will break > if they are not using ``typing.get_type_hints()`` or ``eval()``. > > Annotations that depend on locals at the time of the function > definition will not be resolvable later. Example:: > > def generate(): > A = Optional[int] > class C: > field: A = 1 > def method(self, arg: A) -> None: ... > return C > X = generate() > > Trying to resolve annotations of ``X`` later by using > ``get_type_hints(X)`` will fail because ``A`` and its enclosing scope no > longer exists. Python will make no attempt to disallow such annotations > since they can often still be successfully statically analyzed, which is > the predominant use case for annotations. > > Annotations using nested classes and their respective state are still > valid. They can use local names or the fully qualified name. Example:: > > class C: > field = 'c_field' > def method(self) -> C.field: # this is OK > ... > > def method(self) -> field: # this is OK > ... > > def method(self) -> C.D: # this is OK > ... > > def method(self) -> D: # this is OK > ... > > class D: > field2 = 'd_field' > def method(self) -> C.D.field2: # this is OK > ... > > def method(self) -> D.field2: # this is OK > ... > > def method(self) -> field2: # this is OK > ... > > def method(self) -> field: # this FAILS, class D doesn't > ... # see C's attributes, This was > # already true before this PEP. > > In the presence of an annotation that isn't a syntactically valid > expression, SyntaxError is raised at compile time. However, since names > aren't resolved at that time, no attempt is made to validate whether > used names are correct or not. > > Deprecation policy > ------------------ > > Starting with Python 3.7, a ``__future__`` import is required to use the > described functionality. No warnings are raised. > > In Python 3.8 a ``PendingDeprecationWarning`` is raised by the > compiler in the presence of type annotations in modules without the > ``__future__`` import. > > Starting with Python 3.9 the warning becomes a ``DeprecationWarning``. > > In Python 4.0 this will become the default behavior. Use of annotations > incompatible with this PEP is no longer supported. > > > Forward References > ================== > > Deliberately using a name before it was defined in the module is called > a forward reference. For the purpose of this section, we'll call > any name imported or defined within a ``if TYPE_CHECKING:`` block > a forward reference, too. > > This PEP addresses the issue of forward references in *type annotations*. > The use of string literals will no longer be required in this case. > However, there are APIs in the ``typing`` module that use other syntactic > constructs of the language, and those will still require working around > forward references with string literals. The list includes: > > * type definitions:: > > T = TypeVar('T', bound='<type>') > UserId = NewType('UserId', '<type>') > Employee = NamedTuple('Employee', [('name', '<type>', ('id', > '<type>')]) > > * aliases:: > > Alias = Optional['<type>'] > AnotherAlias = Union['<type>', '<type>'] > YetAnotherAlias = '<type>' > > * casting:: > > cast('<type>', value) > > * base classes:: > > class C(Tuple['<type>', '<type>']): ... > > Depending on the specific case, some of the cases listed above might be > worked around by placing the usage in a ``if TYPE_CHECKING:`` block. > This will not work for any code that needs to be available at runtime, > notably for base classes and casting. For named tuples, using the new > class definition syntax introduced in Python 3.6 solves the issue. > > In general, fixing the issue for *all* forward references requires > changing how module instantiation is performed in Python, from the > current single-pass top-down model. This would be a major change in the > language and is out of scope for this PEP. > > > Rejected Ideas > ============== > > Keeping the ability to use function local state when defining annotations > ------------------------------------------------------------------------- > > With postponed evaluation, this would require keeping a reference to > the frame in which an annotation got created. This could be achieved > for example by storing all annotations as lambdas instead of strings. > > This would be prohibitively expensive for highly annotated code as the > frames would keep all their objects alive. That includes predominantly > objects that won't ever be accessed again. > > To be able to address class-level scope, the lambda approach would > require a new kind of cell in the interpreter. This would proliferate > the number of types that can appear in ``__annotations__``, as well as > wouldn't be as introspectable as strings. > > Note that in the case of nested classes, the functionality to get the > effective "globals" and "locals" at definition time is provided by > ``typing.get_type_hints()``. > > If a function generates a class or a function with annotations that > have to use local variables, it can populate the given generated > object's ``__annotations__`` dictionary directly, without relying on > the compiler. > > Disallowing local state usage for classes, too > ---------------------------------------------- > > This PEP originally proposed limiting names within annotations to only > allow names from the model-level scope, including for classes. The > author argued this makes name resolution unambiguous, including in cases > of conflicts between local names and module-level names. > > This idea was ultimately rejected in case of classes. Instead, > ``typing.get_type_hints()`` got modified to populate the local namespace > correctly if class-level annotations are needed. > > The reasons for rejecting the idea were that it goes against the > intuition of how scoping works in Python, and would break enough > existing type annotations to make the transition cumbersome. Finally, > local scope access is required for class decorators to be able to > evaluate type annotations. This is because class decorators are applied > before the class receives its name in the outer scope. > > Introducing a new dictionary for the string literal form instead > ---------------------------------------------------------------- > > Yury Selivanov shared the following idea: > > 1. Add a new special attribute to functions: ``__annotations_text__``. > > 2. Make ``__annotations__`` a lazy dynamic mapping, evaluating > expressions from the corresponding key in ``__annotations_text__`` > just-in-time. > > This idea is supposed to solve the backwards compatibility issue, > removing the need for a new ``__future__`` import. Sadly, this is not > enough. Postponed evaluation changes which state the annotation has > access to. While postponed evaluation fixes the forward reference > problem, it also makes it impossible to access function-level locals > anymore. This alone is a source of backwards incompatibility which > justifies a deprecation period. > > A ``__future__`` import is an obvious and explicit indicator of opting > in for the new functionality. It also makes it trivial for external > tools to recognize the difference between a Python files using the old > or the new approach. In the former case, that tool would recognize that > local state access is allowed, whereas in the latter case it would > recognize that forward references are allowed. > > Finally, just-in-time evaluation in ``__annotations__`` is an > unnecessary step if ``get_type_hints()`` is used later. > > Dropping annotations with -O > ---------------------------- > > There are two reasons this is not satisfying for the purpose of this > PEP. > > First, this only addresses runtime cost, not forward references, those > still cannot be safely used in source code. A library maintainer would > never be able to use forward references since that would force the > library users to use this new hypothetical -O switch. > > Second, this throws the baby out with the bath water. Now *no* runtime > annotation use can be performed. PEP 557 is one example of a recent > development where evaluating type annotations at runtime is useful. > > All that being said, a granular -O option to drop annotations is > a possibility in the future, as it's conceptually compatible with > existing -O behavior (dropping docstrings and assert statements). This > PEP does not invalidate the idea. > > Pass string literals in annotations verbatim to ``__annotations__`` > ------------------------------------------------------------------- > > This PEP originally suggested directly storing the contents of a string > literal under its respective key in ``__annotations__``. This was > meant to simplify support for runtime type checkers. > > Mark Shannon pointed out this idea was flawed since it wasn't handling > situations where strings are only part of a type annotation. > > The inconsistency of it was always apparent but given that it doesn't > fully prevent cases of double-wrapping strings anyway, it is not worth > it. > > Make the name of the future import more verbose > ----------------------------------------------- > > Instead of requiring the following import:: > > from __future__ import annotations > > the PEP could call the feature more explicitly, for example > ``string_annotations``, ``stringify_annotations``, > ``annotation_strings``, ``annotations_as_strings``, ``lazy_anotations``, > ``static_annotations``, etc. > > The problem with those names is that they are very verbose. Each of > them besides ``lazy_annotations`` would constitute the longest future > feature name in Python. They are long to type and harder to remember > than the single-word form. > > There is precedence of a future import name that sounds overly generic > but in practice was obvious to users as to what it does:: > > from __future__ import division > > > Prior discussion > ================ > > In PEP 484 > ---------- > > The forward reference problem was discussed when PEP 484 was originally > drafted, leading to the following statement in the document: > > A compromise is possible where a ``__future__`` import could enable > turning *all* annotations in a given module into string literals, as > follows:: > > from __future__ import annotations > > class ImSet: > def add(self, a: ImSet) -> List[ImSet]: ... > > assert ImSet.add.__annotations__ == { > 'a': 'ImSet', 'return': 'List[ImSet]' > } > > Such a ``__future__`` import statement may be proposed in a separate > PEP. > > python/typing#400 > ----------------- > > The problem was discussed at length on the typing module's GitHub > project, under `Issue 400 <https://github.com/python/typing/issues/400>`_. > The problem statement there includes critique of generic types requiring > imports from ``typing``. This tends to be confusing to > beginners: > > Why this:: > > from typing import List, Set > def dir(o: object = ...) -> List[str]: ... > def add_friends(friends: Set[Friend]) -> None: ... > > But not this:: > > def dir(o: object = ...) -> list[str]: ... > def add_friends(friends: set[Friend]) -> None ... > > Why this:: > > up_to_ten = list(range(10)) > friends = set() > > But not this:: > > from typing import List, Set > up_to_ten = List[int](range(10)) > friends = Set[Friend]() > > While typing usability is an interesting problem, it is out of scope > of this PEP. Specifically, any extensions of the typing syntax > standardized in PEP 484 will require their own respective PEPs and > approval. > > Issue 400 ultimately suggests postponing evaluation of annotations and > keeping them as strings in ``__annotations__``, just like this PEP > specifies. This idea was received well. Ivan Levkivskyi supported > using the ``__future__`` import and suggested unparsing the AST in > ``compile.c``. Jukka Lehtosalo pointed out that there are some cases > of forward references where types are used outside of annotations and > postponed evaluation will not help those. For those cases using the > string literal notation would still be required. Those cases are > discussed briefly in the "Forward References" section of this PEP. > > The biggest controversy on the issue was Guido van Rossum's concern > that untokenizing annotation expressions back to their string form has > no precedent in the Python programming language and feels like a hacky > workaround. He said: > > One thing that comes to mind is that it's a very random change to > the language. It might be useful to have a more compact way to > indicate deferred execution of expressions (using less syntax than > ``lambda:``). But why would the use case of type annotations be so > all-important to change the language to do it there first (rather > than proposing a more general solution), given that there's already > a solution for this particular use case that requires very minimal > syntax? > > Eventually, Ethan Smith and schollii voiced that feedback gathered > during PyCon US suggests that the state of forward references needs > fixing. Guido van Rossum suggested coming back to the ``__future__`` > idea, pointing out that to prevent abuse, it's important for the > annotations to be kept both syntactically valid and evaluating correctly > at runtime. > > First draft discussion on python-ideas > -------------------------------------- > > Discussion happened largely in two threads, `the original announcement > <https://mail.python.org/pipermail/python-ideas/2017- > September/thread.html#47031>`_ > and a follow-up called `PEP 563 and expensive backwards compatibility > <https://mail.python.org/pipermail/python-ideas/2017- > September/thread.html#47108>`_. > > The PEP received rather warm feedback (4 strongly in favor, > 2 in favor with concerns, 2 against). The biggest voice of concern on > the former thread being Steven D'Aprano's review stating that the > problem definition of the PEP doesn't justify breaking backwards > compatibility. In this response Steven seemed mostly concerned about > Python no longer supporting evaluation of annotations that depended on > local function/class state. > > A few people voiced concerns that there are libraries using annotations > for non-typing purposes. However, none of the named libraries would be > invalidated by this PEP. They do require adapting to the new > requirement to call ``eval()`` on the annotation with the correct > ``globals`` and ``locals`` set. > > This detail about ``globals`` and ``locals`` having to be correct was > picked up by a number of commenters. Nick Coghlan benchmarked turning > annotations into lambdas instead of strings, sadly this proved to be > much slower at runtime than the current situation. > > The latter thread was started by Jim J. Jewett who stressed that > the ability to properly evaluate annotations is an important requirement > and backwards compatibility in that regard is valuable. After some > discussion he admitted that side effects in annotations are a code smell > and modal support to either perform or not perform evaluation is > a messy solution. His biggest concern remained loss of functionality > stemming from the evaluation restrictions on global and local scope. > > Nick Coghlan pointed out that some of those evaluation restrictions from > the PEP could be lifted by a clever implementation of an evaluation > helper, which could solve self-referencing classes even in the form of a > class decorator. He suggested the PEP should provide this helper > function in the standard library. > > Second draft discussion on python-dev > ------------------------------------- > > Discussion happened mainly in the `announcement thread < > https://mail.python.org/pipermail/python-dev/2017-November/150062.html>`_, > followed by a brief discussion under `Mark Shannon's post > <https://mail.python.org/pipermail/python-dev/2017-November/150637.html > >`_. > > Steven D'Aprano was concerned whether it's acceptable for typos to be > allowed in annotations after the change proposed by the PEP. Brett > Cannon responded that type checkers and other static analyzers (like > linters or programming text editors) will catch this type of error. > Jukka Lehtosalo added that this situation is analogous to how names in > function bodies are not resolved until the function is called. > > A major topic of discussion was Nick Coghlan's suggestion to store > annotations in "thunk form", in other words as a specialized lambda > which would be able to access class-level scope (and allow for scope > customization at call time). He presented a possible design for it > (`indirect attribute cells > <https://mail.python.org/pipermail/python-dev/2017-November/150141.html > >`_). > This was later seen as equivalent to "special forms" in Lisp. Guido van > Rossum expressed worry that this sort of feature cannot be safely > implemented in twelve weeks (i.e. in time before the Python 3.7 beta > freeze). > > After a while it became clear that the point of division between > supporters of the string form vs. supporters of the thunk form is > actually about whether annotations should be perceived as a general > syntactic element vs. something tied to the type checking use case. > > Finally, Guido van Rossum declared he's rejecting the thunk idea > based on the fact that it would require a new building block in the > interpreter. This block would be exposed in annotations, multiplying > possible types of values stored in ``__annotations__`` (arbitrary > objects, strings, and now thunks). Moreover, thunks aren't as > introspectable as strings. Most importantly, Guido van Rossum > explicitly stated interest in gradually restricting the use of > annotations to static typing (with an optional runtime component). > > Nick Coghlan got convinced to PEP 563, too, promptly beginning > the mandatory bike shedding session on the name of the ``__future__`` > import. Many debaters agreed that ``annotations`` seems like > an overly broad name for the feature name. Guido van Rossum briefly > decided to call it ``string_annotations`` but then changed his mind, > arguing that ``division`` is a precedent of a broad name with a clear > meaning. > > The final improvement to the PEP suggested in the discussion by Mark > Shannon was the rejection of the temptation to pass string literals > through to ``__annotations__`` verbatim. > > A side-thread of discussion started around the runtime penalty of > static typing, with topic like the import time of the ``typing`` > module (which is comparable to ``re`` without dependencies, and > three times as heavy as ``re`` when counting dependencies). > > > Acknowledgements > ================ > > This document could not be completed without valuable input, > encouragement and advice from Guido van Rossum, Jukka Lehtosalo, and > Ivan Levkivskyi. > > > Copyright > ========= > > This document has been placed in the public domain. > > > > .. > Local Variables: > mode: indented-text > indent-tabs-mode: nil > sentence-end-double-space: t > fill-column: 70 > coding: utf-8 > End: > > > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > guido%40python.org > > -- --Guido van Rossum (python.org/~guido)
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
