[Python-Dev] Postponed annotations break inspection of dataclasses

2018-09-22 Thread David Hagen 
The new postponed annotations have an unexpected interaction with
dataclasses. Namely, you cannot get the type hints of any of the data
classes methods.

For example, I have some code that inspects the type parameters of a
class's `__init__` method. (The real use case is to provide a default
serializer for the class, but that is not important here.)

```
from dataclasses import dataclass
from typing import get_type_hints

class Foo:
pass

@dataclass
class Bar:
foo: Foo

print(get_type_hints(Bar.__init__))
```

In Python 3.6 and 3.7, this does what is expected; it prints `{'foo':
, 'return': }`.

However, if in Python 3.7, I add `from __future__ import annotations`, then
this fails with an error:

```
NameError: name 'Foo' is not defined
```

I know why this is happening. The `__init__` method is defined in the
`dataclasses` module which does not have the `Foo` object in its
environment, and the `Foo` annotation is being passed to `dataclass` and
attached to `__init__` as the string `"Foo"` rather than as the original
object `Foo`, but `get_type_hints` for the new annotations only does a name
lookup in the module where `__init__` is defined not where the annotation
is defined.

I know that the use of lambdas to implement PEP 563 was rejected for
performance reasons. I could be wrong, but I think this was motivated by
variable annotations because the lambda would have to be constructed each
time the function body ran. I was wondering if I could motivate storing the
annotations as lambdas in class bodies and function signatures, in which
the environment is already being captured and is code that usually only
runs once.
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Postponed annotations break inspection of dataclasses

2018-09-22 Thread Guido van Rossum 
This is a good catch -- thanks for bringing it up. I'm adding Eric Smith
(author of dataclasses) and Ivan Levkivskyi (co-author of typing) as well
as Łukasz Langa (author of PEP 563) to the thread to see if they have
further insights.

Personally I don't think it's feasible to change PEP 563 to use lambdas (if
it were even advisable, which would be a long discussion), but I do think
we might be able to make small improvements to the dataclasses and/or
typing modules to make sure your use case works.

Probably a bugs.python.org issue is a better place to dive into the details
than python-dev.

Thanks again,

--Guido (top-poster in chief)

On Sat, Sep 22, 2018 at 8:32 AM David Hagen  wrote:

> The new postponed annotations have an unexpected interaction with
> dataclasses. Namely, you cannot get the type hints of any of the data
> classes methods.
>
> For example, I have some code that inspects the type parameters of a
> class's `__init__` method. (The real use case is to provide a default
> serializer for the class, but that is not important here.)
>
> ```
> from dataclasses import dataclass
> from typing import get_type_hints
>
> class Foo:
> pass
>
> @dataclass
> class Bar:
> foo: Foo
>
> print(get_type_hints(Bar.__init__))
> ```
>
> In Python 3.6 and 3.7, this does what is expected; it prints `{'foo':
> , 'return': }`.
>
> However, if in Python 3.7, I add `from __future__ import annotations`,
> then this fails with an error:
>
> ```
> NameError: name 'Foo' is not defined
> ```
>
> I know why this is happening. The `__init__` method is defined in the
> `dataclasses` module which does not have the `Foo` object in its
> environment, and the `Foo` annotation is being passed to `dataclass` and
> attached to `__init__` as the string `"Foo"` rather than as the original
> object `Foo`, but `get_type_hints` for the new annotations only does a name
> lookup in the module where `__init__` is defined not where the annotation
> is defined.
>
> I know that the use of lambdas to implement PEP 563 was rejected for
> performance reasons. I could be wrong, but I think this was motivated by
> variable annotations because the lambda would have to be constructed each
> time the function body ran. I was wondering if I could motivate storing the
> annotations as lambdas in class bodies and function signatures, in which
> the environment is already being captured and is code that usually only
> runs once.
> ___
> Python-Dev mailing list
> [email protected]
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>


-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Postponed annotations break inspection of dataclasses

2018-09-22 Thread Eric V. Smith 

On 9/22/2018 12:41 PM, Guido van Rossum wrote:
This is a good catch -- thanks for bringing it up. I'm adding Eric Smith 
(author of dataclasses) and Ivan Levkivskyi (co-author of typing) as 
well as Łukasz Langa (author of PEP 563) to the thread to see if they 
have further insights.


I don't see Ivan and Łukasz cc'd, so I'm adding them here.

Personally I don't think it's feasible to change PEP 563 to use lambdas 
(if it were even advisable, which would be a long discussion), but I do 
think we might be able to make small improvements to the dataclasses 
and/or typing modules to make sure your use case works.


Probably a bugs.python.org  issue is a better 
place to dive into the details than python-dev.


Agreed that opening a bug would be good.

And then I'll ruin that suggestion by answering here, too:

I think this problem is endemic to get_type_hints(). I've never 
understood how you're supposed to use the globals and locals arguments 
to it, but this works:


print(get_type_hints(Bar.__init__, globals()))

as does:

print(get_type_hints(Bar.__init__, Bar.__module__))

But that seems like you'd have to know a lot about how a class were 
declared in order to call get_type_hints on it. I'm not sure __module__ 
is always correct (but again, I haven't really thought about it).


The docs for get_type_hints() says: "In addition, forward references 
encoded as string literals are handled by evaluating them in globals and 
locals namespaces."


Every once in a while someone will bring up the idea of delayed 
evaluation, and the answer is always "use a lambda". If we ever wanted 
to do something more with delayed evaluation, this is a good use case 
for it.


Eric



Thanks again,

--Guido (top-poster in chief)

On Sat, Sep 22, 2018 at 8:32 AM David Hagen > wrote:


The new postponed annotations have an unexpected interaction with
dataclasses. Namely, you cannot get the type hints of any of the
data classes methods.

For example, I have some code that inspects the type parameters of a
class's `__init__` method. (The real use case is to provide a
default serializer for the class, but that is not important here.)

```
from dataclasses import dataclass
from typing import get_type_hints

class Foo:
     pass

@dataclass
class Bar:
     foo: Foo

print(get_type_hints(Bar.__init__))
```

In Python 3.6 and 3.7, this does what is expected; it prints
`{'foo': , 'return': }`.

However, if in Python 3.7, I add `from __future__ import
annotations`, then this fails with an error:

```
NameError: name 'Foo' is not defined
```

I know why this is happening. The `__init__` method is defined in
the `dataclasses` module which does not have the `Foo` object in its
environment, and the `Foo` annotation is being passed to `dataclass`
and attached to `__init__` as the string `"Foo"` rather than as the
original object `Foo`, but `get_type_hints` for the new annotations
only does a name lookup in the module where `__init__` is defined
not where the annotation is defined.

I know that the use of lambdas to implement PEP 563 was rejected for
performance reasons. I could be wrong, but I think this was
motivated by variable annotations because the lambda would have to
be constructed each time the function body ran. I was wondering if I
could motivate storing the annotations as lambdas in class bodies
and function signatures, in which the environment is already being
captured and is code that usually only runs once.
___
Python-Dev mailing list
[email protected] 
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/guido%40python.org



--
--Guido van Rossum (python.org/~guido )


___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/eric%2Ba-python-dev%40trueblade.com


___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Postponed annotations break inspection of dataclasses

2018-09-22 Thread Guido van Rossum 
On Sat, Sep 22, 2018 at 11:29 AM Eric V. Smith  wrote:

> I think this problem is endemic to get_type_hints(). I've never
> understood how you're supposed to use the globals and locals arguments
> to it, but this works:
>
> print(get_type_hints(Bar.__init__, globals()))
>
> as does:
>
> print(get_type_hints(Bar.__init__, Bar.__module__))
>
> But that seems like you'd have to know a lot about how a class were
> declared in order to call get_type_hints on it. I'm not sure __module__
> is always correct (but again, I haven't really thought about it).
>

Still, I wonder if there's a tweak possible of the globals and locals used
when exec()'ing the function definitions in dataclasses.py, so that
get_type_hints() gets the right globals for this use case.

It's really tough to be at the intersection of three PEPs...

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Postponed annotations break inspection of dataclasses

2018-09-22 Thread Yury Selivanov 
On Sat, Sep 22, 2018 at 3:11 PM Guido van Rossum  wrote:
[..]
> Still, I wonder if there's a tweak possible of the globals and locals used 
> when exec()'ing the function definitions in dataclasses.py, so that 
> get_type_hints() gets the right globals for this use case.
>
> It's really tough to be at the intersection of three PEPs...

If it's possible to fix exec() to accept any Mapping (not just dicts),
then we can create a proxy mapping for "Dataclass.__init__.__module__"
module and everything would work as expected.

Here's a very hack-ish fix we can use in meanwhile (even in 3.7.1?):
https://gist.github.com/1st1/37fdd3cc84cd65b9af3471b935b722df

Yury
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com