[issue44926] typing.get_type_hints() raises for type aliases with forward references

[Łukasz Langa]

Łukasz Langa  added the comment:


New changeset 9ffe582a018a2babd14e874ae2b169370633fe33 by Miss Islington (bot) 
in branch '3.10':
bpo-44926: `get_type_hints`: Add note about type aliases with forward refs 
(GH-27859) (GH-27860)
https://github.com/python/cpython/commit/9ffe582a018a2babd14e874ae2b169370633fe33


--
nosy: +lukasz.langa

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44926] typing.get_type_hints() raises for type aliases with forward references

[Guido van Rossum]

Guido van Rossum  added the comment:


New changeset 16b9be4861e007ad483611ba0479feb2b90ea783 by Maximilian Hils in 
branch 'main':
bpo-44926: `get_type_hints`: Add note about type aliases with forward refs 
(#27859)
https://github.com/python/cpython/commit/16b9be4861e007ad483611ba0479feb2b90ea783


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44926] typing.get_type_hints() raises for type aliases with forward references

[miss-islington]

Change by miss-islington :


--
nosy: +miss-islington
nosy_count: 3.0 -> 4.0
pull_requests: +26318
pull_request: https://github.com/python/cpython/pull/27860

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44926] typing.get_type_hints() raises for type aliases with forward references

[Maximilian Hils]

Change by Maximilian Hils :


--
keywords: +patch
pull_requests: +26317
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/27859

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44926] typing.get_type_hints() raises for type aliases with forward references

[Ken Jin]

Ken Jin  added the comment:

> Ken Jin, can you guide Maximilian towards a successful doc update PR?

It seems that Maximilian has already made some contributions to CPython, so I'm 
sure he's somewhat familiar with our workflow :). Nonetheless, @Maximilian if 
you need any help, please do ping me, I'll be happy to.

We could add a ..note: here 
https://docs.python.org/3/library/typing.html#typing.get_type_hints. The 
document is at 
https://github.com/python/cpython/blob/main/Doc/library/typing.rst. The Python 
docs style guide is at https://devguide.python.org/documenting/#style-guide.

Thanks Max for your interest in improving CPython!

PS: I've changed the affected versions to the ones we still bugfix (3.9 and up).

--
versions: +Python 3.11 -Python 3.7, Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44926] typing.get_type_hints() raises for type aliases with forward references

[Guido van Rossum]

Guido van Rossum  added the comment:

We could mention this in the docs for one or more of the following:

- type aliases (old or new syntax)
- forward references
- get_type_hints()

Ken Jin, can you guide Maximilian towards a successful doc update PR?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44926] typing.get_type_hints() raises for type aliases with forward references

[Maximilian Hils]

Maximilian Hils  added the comment:

Thanks Guido! I agree on not pursuing the List["Foo"] case for the reasons you 
mentioned.

Let me know if you think it'd be useful to mention this limitation briefly in 
one of the relevant PEPs or somewhere else. I'm not sure if it meets the bar 
for notability, you probably have a better gut feeling for this.

Other than that I'd propose we close this here as wontfix. Thank you again for 
the very useful feedback! :)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44926] typing.get_type_hints() raises for type aliases with forward references

[Guido van Rossum]

Guido van Rossum  added the comment:

Maximilian, I think your last analysis is spot on. The problem is specific to 
type aliases and how at runtime they do not carry information about scope with 
them. The current type alias syntax (both the original form and the form using 
"x: TypeAlias = ...") was designed for static analysis only.

I suppose we could fix cases like

A = List["Foo"]

but we can't fix for example

A: TypeAlias = "Foo"

since at runtime this just ends up creating a variable A whose value is the 
string "Foo", and if you import and use that in another module, all you have is 
the value "Foo".

I think we have to accept this as a limitation of type aliases when combined 
with runtime access to types (it's not just get_type_hints(), but any mechanism 
that introspects types at runtime).

I'm reluctant to fix the List["Foo"] case, if only because that case is being 
phased out in favor of list["Foo"] anyway, and because it depends on 
typing.List being special -- we can't do the same kind of fixup for 
user-defined classes (e.g. C["Foo"]).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44926] typing.get_type_hints() raises for type aliases with forward references

[Maximilian Hils]

Maximilian Hils  added the comment:

Thank you Ken and Guido for the super quick feedback!

> It's not fine if it were to look in the wrong namespace

That's a really good point. Here's another terrible example:


foo.py:
```
import typing

FooData: typing.TypeAlias = "Data"

class Data:
pass

```

bar.py:
```
import typing
import foo

BarData: typing.TypeAlias = "Data"

def func1(x: foo.FooData, y: BarData):
pass

class Data:
pass

print(typing.get_type_hints(func1))  # incorrectly uses bar.Data twice.

```


I don't see how we could distinguish FooData from BarData without static 
analysis. Changing get_type_hints to not pick the unrelated class with the same 
name would essentially mean not using func.__globals__, which breaks almost all 
ForwardRef evaluation. This doesn't seem viable.

Potential alternatives:

1. Accept the current state: get_type_hints does not work well with type 
aliases that use forward references.
2. Fix it at least for cases where a ForwardRef is constructed immediately 
(typing.List["Foo"], but not "Foo" or list["Foo"]), similar to how it's been 
done for #41249. I have made a very basic proof-of-concept at 
https://github.com/mhils/cpython/commit/4adbcf088d2857166b579f7dd2954ff9981fc7db,
 but it's a mess (see commit comments).
3. Deprecate/discourage use of forward references in type aliases and/or change 
the syntax for [forward references in] type aliases so that their origin can be 
tracked.

In summary, it seems like there are no really good solutions here. I'm fine if 
this ends up as a pragmatic wontfix, maybe with a comment added somewhere in 
the docs or in PEP 613.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44926] typing.get_type_hints() raises for type aliases with forward references

[Guido van Rossum]

Guido van Rossum  added the comment:

I think it's fine for get_type_hints() to give up in some cases. It's not fine 
if it were to look in the wrong namespace (e.g. if the caller of 
get_type_hints() were to happen to have an unrelated class named "Foo" in its 
namespace, that should *not* be found if the type alias was defined in a 
different file).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44926] typing.get_type_hints() raises for type aliases with forward references

[Maximilian Hils]

Maximilian Hils  added the comment:

@Guido van Rossum: Yes, GH-27017 is the same as #41249 in the initial post. 
There are also some cases where we don't even have a ForwardRef though:

foo.py:
```
import typing

FooType: typing.TypeAlias = "Foo"

class Foo:
pass

```

bar.py:
```
import typing
import foo

def func3(x: foo.FooType):
pass

print(typing.get_type_hints(func3))  # NameError: name 'Foo' is not defined.

```

In this example, `FooType` is just a regular str (with unknown origin). That's 
a case where get_type_hints just needs to give up?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44926] typing.get_type_hints() raises for type aliases with forward references

[Guido van Rossum]

Guido van Rossum  added the comment:

Is GH-27017 at all relevant here?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44926] typing.get_type_hints() raises for type aliases with forward references

[Maximilian Hils]

Maximilian Hils  added the comment:

> For your specific use case (where the user is using Python 3.6), you could 
> pass in globalns and localns to get_type_hints as a temporary workaround. Off 
> the top of my head:
get_type_hints(func2, globalns=foo.__dict__) might work. Would that work for 
your library?

I guess the hard part is knowing that the type annotation comes from `foo`. In 
the example here we can of course hardcode it, but that doesn't work in the 
general case or for pdoc, the documentation generator I'm working on 
(https://pdoc.dev). I have experimented quite a bit with walking the AST to 
figure out where type aliases are imported from to then re-executing 
ForwardRefs with that globalns. Long story short, trying to reverse-engineer 
__forward_module__ quickly becomes a tangled hot mess where you need to adjust 
for import aliases, reimports, and so on.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44926] typing.get_type_hints() raises for type aliases with forward references

[Ken Jin]

Ken Jin  added the comment:

> 1. func1 doesn't crash, but also doesn't resolve the forward reference. I am 
> not sure if this expected behavior.

Good observation! That's indeed the current behavior. The reason is a little 
subtle - list[...] is using a special builtin type 
(https://docs.python.org/3/library/stdtypes.html#types-genericalias), while 
typing.List is the one from typing. The typing version wraps all strings in 
ForwardRef(), while the builtin version shouldn't import from typing, so it 
doesn't have that luxury. get_type_hints could probably support the builtin 
version, though I'd imagine we'd need some rework.

Re: 2. You're right there too. Storing the defined module in __forward_module__ 
is an interesting proposal.

For your specific use case (where the user is using Python 3.6), you could pass 
in globalns and localns to get_type_hints as a temporary workaround. Off the 
top of my head:
get_type_hints(func2, globalns=foo.__dict__) might work. Would that work for 
your library?

--
nosy: +gvanrossum, kj

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44926] typing.get_type_hints() raises for type aliases with forward references

[Maximilian Hils]

New submission from Maximilian Hils :

Someone reported this rather interesting issue where typing.get_type_hints 
crashes on type aliases with forward references. The original report is at 
https://github.com/mitmproxy/pdoc/issues/290. Here's an extended minimal 
example:

foo.py:
```
import typing

FooList1: typing.TypeAlias = list["Foo"]
FooList2: typing.TypeAlias = typing.List["Foo"]

class Foo:
pass

```

bar.py:
```
import typing
import foo

def func1(x: foo.FooList1):
pass

def func2(x: foo.FooList2):
pass

print(typing.get_type_hints(func1))  # {'x': list['Foo']}
print(typing.get_type_hints(func2))  # NameError: name 'Foo' is not defined.

```


Observations:

1. func1 doesn't crash, but also doesn't resolve the forward reference. I am 
not sure if this expected behavior.
   If it isn't, this should eventually run in the same problem as func2.
2. func2 crashes because "Foo" is evaluated in the context of bar.py (where 
class Foo is unknown) and not in the context of foo.py. ForwardRef._evaluate 
would
   somehow need to know in which context it was defined. #41249 (TypedDict 
inheritance doesn't work with get_type_hints) introduced 
   ForwardRef.__forward_module__, which would be a logical place for that 
information. I'm not sure if it is a good idea
   to use __forward_module__ more widely.
3. This may end up as quite a bit of complexity for an edge case, I'm fine if 
it is considered wontfix. 
   The reason I'm bringing it up is that PEP 613 (Explicit Type Aliases) 
decidedly allows forward references in type aliases.


For the record, PEP 563 (postponed evaluations) does not change the outcome 
here. However, postponed evaluations often make it possible to avoid the 
forward references by declaring the aliases last.

--
components: Library (Lib)
messages: 399660
nosy: mhils
priority: normal
severity: normal
status: open
title: typing.get_type_hints() raises for type aliases with forward references
type: crash
versions: Python 3.10, Python 3.7, Python 3.8, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com