[
https://issues.apache.org/jira/browse/BEAM-7060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16849612#comment-16849612
]
Robert Bradshaw commented on BEAM-7060:
---------------------------------------
I don't think this will be sufficient. For example, a PTransform may
be parameterized by an SQL statement where the output type(s) would
then depend on the input types(s) and the statement itself. It's also
hard to type things like hetrogenious dictionaries. This kind of thing
forces us to do more typechecking at runtime (but, fortunately, before
the pipeline is actually launched, so in some sense we have a phase
that sits between the traditional static and runtime phases). This is
why BeamJava has moved away from leveraging the static typechecker to
doing its own. But for many basic cases, being able to lift the types
in such a way that a static checker could understand would be great,
e.g. inferring that beam.Map(lambda (x: T) -> O: ...) is a
PTranform[PCollection[T], PCollection[O]]).
I am in total agreement about wanting to avoid re-inventing the world.
Do you know if it's possible to extract any of mypy's
type-compatibility validation or inference code as a library?
+1.
> Design Py3-compatible typehints annotation support in Beam 3.
> -------------------------------------------------------------
>
> Key: BEAM-7060
> URL: https://issues.apache.org/jira/browse/BEAM-7060
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
> Reporter: Valentyn Tymofieiev
> Assignee: Udi Meiri
> Priority: Major
>
> Existing [Typehints implementaiton in
> Beam|[https://github.com/apache/beam/blob/master/sdks/python/apache_beam/typehints/
> ] heavily relies on internal details of CPython implementation, and some of
> the assumptions of this implementation broke as of Python 3.6, see for
> example: https://issues.apache.org/jira/browse/BEAM-6877, which makes
> typehints support unusable on Python 3.6 as of now. [Python 3 Kanban
> Board|https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=245&view=detail]
> lists several specific typehints-related breakages, prefixed with "TypeHints
> Py3 Error".
> We need to decide whether to:
> - Deprecate in-house typehints implementation.
> - Continue to support in-house implementation, which at this point is a stale
> code and has other known issues.
> - Attempt to use some off-the-shelf libraries for supporting
> type-annotations, like Pytype, Mypy, PyAnnotate.
> WRT to this decision we also need to plan on immediate next steps to unblock
> adoption of Beam for Python 3.6+ users. One potential option may be to have
> Beam SDK ignore any typehint annotations on Py 3.6+.
> cc: [~udim], [~altay], [~robertwb].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
