On Fri, Jan 15, 2021 at 10:53 AM Larry Hastings <la...@hastings.org> wrote:
>
> Sorry it took me 3+ days to reply--I had a lot to think about here. But I
> have good things to report!
>
>
> On 1/11/21 8:42 PM, Guido van Rossum wrote:
>
> On Mon, Jan 11, 2021 at 1:20 PM Larry Hastings <la...@hastings.org> wrote:
>
>> PEP 563 states:
>>
>> For code that uses type hints, the typing.get_type_hints(obj,
>> globalns=None, localns=None) function correctly evaluates expressions back
>> from its string form.
>>
>> So, if you are passing in a localns argument that isn't None, okay, but
>> you're not using them "correctly" according to the language. Also, this
>> usage won't be compatible with static type checkers.
>>
> I think you're misreading PEP 563 here. The mention of globalns=None,
> localns=None refers to the fact that these parameters have defaults, not
> that you must pass None. Note that the next paragraph in that PEP mentions
> eval(ann, globals, locals) -- it doesn't say eval(ann, {}, {}).
>
> I think that's misleading, then. The passage is telling you how to
> "correctly evaluate[s] expressions", and how I read it was, it's telling me
> I have to supply globalns=None and localns=None for it to work
> correctly--which, I had to discover on my own, were the default values. I
> don't understand why PEP 563 feels compelled to define a function that it's
> not introducing, and in fact had already shipped with Python two versions
> ago.
>
I suppose PEP 563 is ambiguous because on the one hand global symbols are
the only things that work out of the box, on the other hand you can make
other things work by passing the right scope (and there's lots of code now
that does so), and on the third hand, it claims that get_type_hints() adds
the class scope, which nobody noticed or implemented until this week
(there's a PR, can't recall the number).
But I think all this is irrelevant given what comes below.
>
> Later in that same section, PEP 563 points out a problem with annotations
> that reference class-scoped variables, and claims that the implementation
> would run into problems because methods can't "see" the class scope. This
> is indeed a problem for PEP 563, but *you* can easily generate correct
> code, assuming the containing class exists in the global scope (and your
> solution requires that anyway). So in this case
> ```
> class Outer:
> class Inner:
> ...
> def method(self, a: Inner, b: Outer) -> None:
> ...
> ```
> The generated code for the `__annotations__` property could just have a
> reference to `Outer.Inner` for such cases:
> ```
> def __annotations__():
> return {"a": Outer.Inner, "b": Outer, "return": None}
> ```
>
> This suggestion was a revelation for me. Previously, a combination of bad
> experiences early on when hacking on compile and symtable, and my
> misunderstanding of exactly what was being asserted in the November 2017
> thread, led me to believe that all I could support was globals. But I've
> been turning this over in my head for several days now, and I suspect I can
> support... just about anything.
>
>
> I can name five name resolution scenarios I might encounter. I'll discuss
> them below, in increasing order of difficulty.
>
>
> *First* is references to globals / builtins. That's already working,
> it's obvious how it works, and I need not elaborate further.
>
Yup.
>
> *Second* is local variables in an enclosing function scope:
>
> def outer_fn():
> class C: pass
> def inner_fn(a:C=None): pass
> return inner_fn
>
> As you pointed out elsewhere in un-quoted text, I could make the
> annotation a closure, so it could retain a reference to the value of (what
> is from its perspective) the free variable "C".
>
Yup.
>
> *Third* is local variables in an enclosing class scope, as you describe
> above:
>
> class OuterCls:
> class InnerCls:
> def method(a:InnerCls=None): pass
>
> If I understand what you're suggesting, I could notice inside the compiler
> that Inner is being defined in a class scope, walk up the enclosing scopes
> until I hit the outermost class, then reconstruct the chain of pulling out
> attributes until it resolves globally. Thus I'd rewrite this example to:
>
> class OuterCls:
> class InnerCls:
> def method(a:OuterCls.InnerCls=None): pass
>
> We've turned the local reference into a global reference, and we already
> know globals work fine.
>
I think this is going too far. A static method defined in InnerCls does not
see InnerCls (even after the class definitions are complete). E.g.
```
class Outer:
class Inner:
@staticmethod
def foo(): return Inner
```
If you then call Outer.Inner.foo() you get "NameError: name 'Inner' is not
defined".
>
> *Fourth* is local variables in an enclosing class scope, which are
> themselves local variables in an enclosing function scope:
>
> def outerfn():
> class OuterCls:
> class InnerCls:
> def method(a:InnerCls=None): pass
> return OuterCls.InnerCls
>
> Even this is solvable, I just need to combine the "second" and "third"
> approaches above. I walk up the enclosing scopes to find the outermost
> class scope, and if that's a function scope, I create a closure and retain
> a reference to *that* free variable. Thus this would turn into
>
> def outerfn():
> class OuterCls:
> class InnerCls:
> def method(a:OuterCls.InnerCls=None): pass
>
> and method.__co_annotations__ would reference the free variable "OuterCls"
> defined in outerfn.
>
Probably also not needed.
>
> *Fifth* is the nasty one. Note that so far every definition we've
> referred to in an annotation has been *before* the definition of the
> annotation. What if we want to refer to something defined *after* the
> annotation?
>
> def outerfn():
> class OuterCls:
> class InnerCls:
> def method(a:zebra=None): pass
> ...
>
> We haven't seen the definition of "zebra" yet, so we don't know what
> approach to take. It could be any of the previous four scenarios. What do
> we do?
>
If you agree with me that (3) and (4) are unnecessary (or even
undesirable), the options here are either that zebra is a local in
outerfn() (then just make it a closure), and if it isn't you should treat
it as a global.
> This is solvable too: we simply delay the compilation of
> __co_annotations__ code objects until the very last possible moment.
> First, at the time we bind the class or function, we generate a stub
> __co_annotations__ object, just to give the compiler what it expects. The
> compiler inserts it into the const table for the enclosing construct
> (function / class / module), and we remember what index it went into.
> Then, after we've finished processing the entire AST tree for this module,
> but before we we exit the compiler, we reconstruct the required context for
> evaluating each __co_annotations__ function--the nested chain of symbol
> tables, the compiler blocks if needed, etc--and evaluate the annotations
> for real. We assemble the correct __co_annotations__ code object and
> overwrite the stub in the const table with this now-correct value.
>
> I can't think of any more scenarios. So, I think I can handle basically
> anything!
>
>
> However, there are two scenarios where the behavior of evaluations will
> change in a way the user might find surprising. The first is when they
> redefine a variable used in an annotation:
>
> x = str
> def fn(a:x="345"): pass
> x = int
>
> With stock semantics, the annotation to "a" will be "str". With PEP 563
> or my PEP, the annotation to "a" will be "int". (It gets even more
> exciting if you said "del x".)
>
This falls under the Garbage in, Garbage out principle. Mypy doesn't even
let you do this. Another type checker which is easy to install, pyright,
treats it as str. I wouldn't worry too much about it. If you strike the
first definition of x, the pyright complains and mypy treats it as int.
> Similarly, delaying the annotations so that we make everything visible
> means defining variables with the same name in multiple scopes may lead to
> surprising behavior.
>
> x = str
> class Outer:
> def method(a:x="345"): pass
> x = int
>
> Again, stock gets you an annotation of "str", but PEP 563 and my PEP gets
> you "str", because they'll see the *final* result of evaluating the body
> of Outer.
>
> Sadly this is the price you pay for delayed evaluation of annotations.
> Delaying the evaluation of annotations is the goal, and the whole point is
> to make changes, observable by the user, in how annotations are evaluated.
> All we can do is document these behaviors and hope our users forgive us.
>
Agreed.
>
> I think this is a vast improvement over the first draft of my PEP, and
> assuming nobody points out major flaws in this approach (and, preferably,
> at least a little encouragement), I plan to redesign my prototype along
> these lines. (Though not right away--I want to take a break and attend to
> some other projects first.)
>
>
> Thanks for the mind-blowing suggestions, Guido! I must say, you're pretty
> good at this Python stuff.
>
You're not so bad yourself -- without your wakeup call we would have
immortalized PEP 563's limitations.
--
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*
<http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/ICODQ2JEEXBN67BPEHJRTME6KJF5OQ4M/
Code of Conduct: http://python.org/psf/codeofconduct/
