Thanks for your detailed review!
> On Sep 11, 2017, at 9:45 PM, Steven D'Aprano <st...@pearwood.info> wrote:
>
> Regarding forward references: I see no problem with quoting forward
> references. Some people complain about the quotation marks, but frankly
> I don't think that's a major imposition.
Are you a user of type annotations? In the introduction of typing at Facebook
this is the single most annoying thing people point out. The reason is that
it's not obvious and the workaround is ugly. Learning about this quirk adds to
an already significant body of knowledge new typing users have to familiarize
themselves with. It's also aesthetically jarring, the original PEP 484 admits
this is a clunky solution.
> Regarding the execution time at runtime: this sounds like premature
> optimization. If it is not, if you have profiled some applications found
> that there is more than a trivial amount of time used by generating the
> annotations, you should say so.
I used hand-wavy language because I didn't really check before. This time
around I'm coming back prepared. Instagram has roughly 10k functions annotated
at this point. Using tracemalloc I tested how much memory it takes to import
modules with those functions. Then I compared how much memory it takes to
import modules with those functions when annotations are stripped from the
entire module (incl. module-level variable annotations, aliases, class variable
annotations, etc.).
The difference in allocated memory is over 22 MB. The import time with
annotations is over 2s longer. The problem with those numbers that we still
have 80% functions to cover.
> In my opinion, a less disruptive solution to the execution time
> (supposed?) problem is a switch to disable annotations altogether,
> similar to the existing -O optimize switch, turning them into no-ops.
> That won't make any difference to static type-checkers.
>
> Some people will probably want such a switch anyway, if they care about
> the memory used by the __annotations__ dictionaries.
Yes, this is a possibility that I should address in the PEP explicitly. There
are two reasons this is not satisfying:
1. This only addresses runtime cost, not forward references, those still cannot
be safely used in source code. Even if a single one was used in a single
module, the entire program now has to be executed with this new hypothetical -O
switch. Nobody would agree to dependencies like that.
2. This throws the baby out with the bath water. Now *no* runtime annotation
use can be performed. There's work on new tools that use PEP 484-compliant type
annotations at runtime (Larry is reviving his dryparse library, Eric Smith is
working on data classes). Those would not work with this hypothetical -O switch
but are fine with the string form since they already need to use
`typing.get_type_hints()`.
>> If an annotation was already a string, this string is preserved
>> verbatim. In other cases, the string form is obtained from the AST
>> during the compilation step, which means that the string form preserved
>> might not preserve the exact formatting of the source.
>
> Can you give an example of how the AST may distort the source?
>
> My knee-jerk reaction is that anything which causes the annotation to
> differ from the source is likely to cause confusion.
Anything not explicitly stored in the AST will require a normalized
untokenization. This covers whitespace, punctuation, and brackets. An example:
def fun(a: Dict[
str,
str,
]) -> None: ...
would become something like:
{'a': 'Dict[(str, str)]', 'return': 'None'}
>> Annotations need to be syntactically valid Python expressions, also when
>> passed as literal strings (i.e. ``compile(literal, '', 'eval')``).
>> Annotations can only use names present in the module scope as postponed
>> evaluation using local names is not reliable.
>
> And that's a problem. Forcing the use of global variables seems harmful.
> Preventing the use of locals seems awful. Can't we use some sort of
> closure-like mechanism?
As Nick measured, currently closures would add to the performance problem, not
solve it. What is an actual use case that this would prevent?
> This restriction is going to break, or prevent, situations like this:
>
> def decorator(func):
> kind = ... # something generated at decorator call time
> @functools.wraps(func)
> def inner(arg: kind):
> ...
> return inner
>
>
> Even if static typecheckers have no clue what the annotation on the
> inner function is, it is still useful for introspection and
> documentation.
Do you have a specific use case in mind? If you need to put magic dynamic
annotations for runtime introspection, the decorator can apply those directly
via calling
inner.__annotations__ = {'arg': 'kind'}
This is what the "attrs" library is considering doing for the magic __init__
method. Having annotation literals in the source not validated would not hinder
that at all.
That's it for introspection. As for documentation, I don't really understand
that comment.
>> Resolving Type Hints at Runtime
>> ===============================
> [...]
>> To get the correct module-level
>> context to resolve class variables, use::
>>
>> cls_globals = sys.modules[SomeClass.__module__].__dict__
>
> A small style issue: I think that's better written as:
>
> cls_globals = vars(sys.modules[SomeClass.__module__])
>
> We should avoid directly accessing dunders unless necessary, and vars()
> exists specifically for the purpose of returning object's __dict__.
Right! Would be great if it composed a dictionary of slots for us, too ;-)
>> Runtime annotation resolution and ``TYPE_CHECKING``
>> ---------------------------------------------------
>>
>> Sometimes there's code that must be seen by a type checker but should
>> not be executed. For such situations the ``typing`` module defines a
>> constant, ``TYPE_CHECKING``, that is considered ``True`` during type
>> checking but ``False`` at runtime. Example::
>>
>> import typing
>>
>> if typing.TYPE_CHECKING:
>> import expensive_mod
>>
>> def a_func(arg: expensive_mod.SomeClass) -> None:
>> a_var: expensive_mod.SomeClass = arg
>> ...
>
> I don't know whether this is important, but for the record the current
> documentation shows expensive_mod.SomeClass quoted.
Yes, I wanted to explicitly illustrate that now you can use the annotations
directly without quoting and it won't fail at import time. If you tried to
evaluate them with `typing.get_type_hints()` that would fail, just like trying
to evaluate the string form today.
>> Backwards Compatibility
>> =======================
> [...]
>> Annotations that depend on locals at the time of the function/class
>> definition are now invalid. Example::
>
> As mentioned above, I think this is a bad loss of existing
> functionality.
I don't quite see it.
If somebody badly needs to store arbitrary data as annotations in generated
functions, they still can directly write to `__annotations__`. In every other
case, specifically in static typing (the overwhelming use case for
annotations), this makes human readers happier and static analysis tools are
indifferent.
In fact, if runtime type hint evaluation is necessary, then even if a library
is already using `typing.get_type_hints()` (which it should!), so far I haven't
seen much use of `globalns` and `localns` arguments to this function which
suggests that the forward references we can support are already constrained to
globals anyway.
>> In the presence of an annotation that cannot be resolved using the
>> current module's globals, a NameError is raised at compile time.
>
> It is not clear what this means, or how it will work, especially given
> that the point of this is to delay evaluating annotations. How will the
> compiler know that an annotation cannot be resolved if it doesn't try to
> evaluate it?
The idea would be to validate the expressions after the entire module is
compiled, something like what the flake8-pyi plugin is doing today for .pyi
files. Guido pointed out that it's not trivial since the compiler doesn't keep
a symbol table around. But I'd invest time in this since I agree with your
point that we should raise errors as early as possible.
>> Rejected Ideas
>> ==============
>>
>> Keep the ability to use local state when defining annotations
>> -------------------------------------------------------------
>>
>> With postponed evaluation, this is impossible for function locals.
>
> Impossible seems a bit strong. Can you elaborate?
Unless we stored the closure, it's not possible.
> [...]
>> This is brittle and doesn't even cover slots. Requiring the use of
>> module-level names simplifies runtime evaluation and provides the
>> "one obvious way" to read annotations. It's the equivalent of absolute
>> imports.
>
> I hardly think that "simplifies runtime evaluation" is true. At the
> moment annotations are already evaluated. *Anything* that you have to do
> by hand (like call eval) cannot be simpler than "do nothing".
This section covers the rejected idea of allowing local access on classes
specifically. As I mentioned earlier, preserving local function scope access
is not something we can do without bending ourselves backwards. Constraining
ourselves to only global scope simplifies runtime evaluation compared to that
use case. The alternative is not "do nothing".
> I don't think the analogy with absolute imports is even close to useful,
> and far from being "one obvious way", this is a surprising, non-obvious,
> seemingly-arbitrary restriction on annotations.
That's a bit harsh. I probably didn't voice my idea clearly enough here so let
me elaborate. Annotations inside nested classes which are using local scope
currently have to use the local names directly instead of using the qualified
name. This has similar issues to relative imports:
class Menu(UIComponent):
...
class Restaurant:
class Menu(Enum):
SPAM = 1
EGGS = 2
def generate_menu_component(self) -> Menu:
...
It is impossible today to use the global "Menu" type in the annotation of the
example method. This PEP is proposing to use qualified names in this case which
disambiguates between the global "Menu" and "Restaurant.Menu". In this sense it
felt similar to absolute imports to me.
> Given that for *six versions* of Python 3 annotations could be locals,
> there is nothing obvious about restricting them to globals.
I don't understand the argument of status quo. In those six versions, the first
two were not usable in production and the following two only slowly started
getting adoption. It wasn't until Python 3.5 that we started seeing significant
adoption figures. This is incidentally also the first release with PEP 484.
Wow, this took quite some time to respond to, sorry it took so long! I had to
do additional research and clarify some of my own understanding while doing
this. It was very valuable, thanks again for your feedback!
- Ł
signature.asc
Description: Message signed with OpenPGP
_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
