New submission from Andreas H. <[email protected]>:
(De)Serialization of in-memory data structures is an important application.
However there is a rather unpleasant issue with ForwardRefs.
One cannot export type aliases when they contain ForwardRefs (and expect
things to work).
Consider the example:
Json = Union[ List['Json'], Dict[str, 'Json'], int, float, bool, None ]
When used in another module the containing ForwardRefs cannot be resolved, or
in the worst case, resolve to
wrong types if in the caller namespace contains a symbol with the same name as
ForwardRef refers to.
This of course only applies to inspection-based tools which utilize the
run-time information, not the static type checkers.
Type aliases sometimes have to be used sometimes - they cannot be avoided in
all cases,
especially with recursive types as in the example above.
There are several options to improve the situation. These are all that came to
my mind and I want to expose them to discussion.
1. Guard with NewType
Json = NewType('Json', Union[ List['Json'], Dict[str, 'Json'], int, float,
bool, None ] )
Advantage: This could allow automatic cross-module ForwardRef resolution
provided issue 46369 [1] is acknowledged as a bug and fixed.
Disadvantage: Does not create a true type alias, but a sub-type. Type casts
have to be used all the time, e.g. `data = Json(bool)`.
So can only applied to a subset of use-cases (but is IMO a clean solution
when it fits).
2. Use the `module` parameter of ForwardRef
Json = Union[ List[ForwardRef('Json', module=__name__)], Dict[str,
ForwardRef('Json', module=__name__)], int, float, bool, None ] )
Advantage: Works in 3.10.
Disadvantage: Would require issue 46333 [2] to be fixed. ForwardRef is not
meant to be instatiated by the user,
also `module` parameter is currently completely internal.
3. Modify ForwardRef so that it accepts a fully qualified name
Json = Union[ List[__name__+'.Json'], Dict[str, __name__+'.Json'], int,
float, bool, None ] )
Advantage: This is only a tiny change (because ForwardRef has the `module`
parameter). ForwardRef would stay internal. Less ugly than 2.
Disadvantage: Still a bit ugly. Would also require issue 46333 [2] to be
fixed. Relative module specification (as in relative imports)
would not work.
4. Manual evaluation
Json = Union[ List['Json'], Dict[str, 'Json'], int, float, bool, None ]
resolve_type_alias(Json, globals(), locals() )
def resolve_type_alias(type, globalns, localns):
class dummy:
test: type
typing.get_type_hints(dummy, globalns, localns) # Note: this modifies
ForwardRefs in-place
Advantage: Works in many versions.
Disadvantage: Requires user to explicily call function after the last
referenced type is defined
(this may be physically separated from the type alias definition, which
does not feel like a good solution especially since
this ForwardRef export problem is only technical, and not even close to
beeing obvious to most people)
5. Make `get_type_hints()` to work with type-aliases (essentially integrate
above `resolve_type_alias`). The return value
of get_type_hints() would be the same as argument, just with ForwardRefs
in-place resolved.
Json = Union[ List['Json'], Dict[str, 'Json'], int, float, bool, None ]
get_type_hints(Json, globals(), locals())
Advantage: same as 4) but hides useful (but ugly) code
Disadvantage: same as 4)
6. Make all types in typing (such as List, Dict, Union, etc...) to capture
their calling module and pass this information to ForwardRef, when
one is to be created. Then already during construction of ForwardRef the
`module` will be correctly set.
Json = Union[ List['Json'], Dict[str, 'Json'], int, float, bool, None ]
Advantage: This requires no user intervention. Things will "just work"
Disadvantage: This is rather big change. It is incompatible with the caching
used inside typing.py (the new __module__ parameter would
need to be taken into account in __hash__ and/or __eq__). And maybe has
other issues I do not see at the moment.
7. Extend `TypeAlias` hint so it can be used in bracket way similar to e.g.
`Annotated`
Json = TypeAlias[ Union[ List['Json'], Dict[str, 'Json'], int, float,
bool, None ] ]
I know, presently it is supposed to be used as `Json: TypeAlias = Union[
.... ]`. But that is of no help at run-time, because
the variable Json contains no run-time information. So even if TypeAlias
would capture the calling module,
this information is not passed on to the variable `Json`. This is different
for the bracket notation TypeAlias[ .. ].
Advantage: Similar usage to Annotated. Would not require a big change such
as in 4).
Disadvantage: TypeAlias is not supposed to be used that way.
8. Introduce function to define a type alias, which sets the module parameter
of all ForwardRefs
Json = DefineTypeAlias( Union[ List['Json'], Dict[str, 'Json'], int,
float, bool, None ] )
DefineTypeAlias would capture its calling module and recursively walk over
the type tree to find and patch all ForwardRefs
Advantage: Simpler change, but requires to recurively walk over all alias
type-hints which do not capture their calling module.
Disadvantage: Together with TypeAlias, things look ugly and unnecessary
complicated. (Json: TypeAlias = DefineTypeAlias( ... ) )
Personally, I think 1) is a good solution in cases where this can be applied
(and [1] is considered a bug and fixed).
For the other cases 6) would be the ideal solution, but this may be too much
work. Alternatively I think 3), 8) or 5) (in that order)
may be interesting and could be a potential enhancement to the `typing` module.
I would really appreciate some thoughts or comments on this! Thank you.
- [1] https://bugs.python.org/issue46369
- [2] https://bugs.python.org/issue46333
----------
components: Library (Lib)
messages: 410535
nosy: AlexWaygood, andreash, gvanrossum, kj, kumaraditya303
priority: normal
severity: normal
status: open
title: A better way to resolve ForwardRefs in type aliases across modules
type: enhancement
versions: Python 3.11
_______________________________________
Python tracker <[email protected]>
<https://bugs.python.org/issue46371>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
