[
https://issues.apache.org/jira/browse/BEAM-7060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16850832#comment-16850832
]
Robert Bradshaw commented on BEAM-7060:
---------------------------------------
Consider the case
{code:python}
(p
| beam.io.Read(beam.io.BigQuerySource(table_spec)) # type of the result
determined by querrying bigquery at runtime
| beam.sql.SqpTransform("some statement") # output types inferred from
input types based on contents of sql
| beam.MyTransform()) # may have some type constraints of its own
{code}
I don't know how to reasonably check this statically, but it can be done well
at runtime. Even if I were to add some typehints (how?) so that a tool like
mypy could statically validate this, these hints would become out of date as
the schema of the table or my sql statement (both of which could be passed in
from elsewhere) change or become out of date.
If we can't use mypy as a library, we may have to look around for something
else. Or roll our own. Maybe we could "libraryify" the public API by generating
some code like
{code:python}
def check_compatible(type1, type2):
try:
mypy.cli("""
input : %s
output : %s
output = input
""" % (type1, type2)
except:
raise BetterError()
{code}
but that seems pretty hacky...
(Also, good to know about TypedDict.)
> Design Py3-compatible typehints annotation support in Beam 3.
> -------------------------------------------------------------
>
> Key: BEAM-7060
> URL: https://issues.apache.org/jira/browse/BEAM-7060
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
> Reporter: Valentyn Tymofieiev
> Assignee: Udi Meiri
> Priority: Major
>
> Existing [Typehints implementaiton in
> Beam|[https://github.com/apache/beam/blob/master/sdks/python/apache_beam/typehints/
> ] heavily relies on internal details of CPython implementation, and some of
> the assumptions of this implementation broke as of Python 3.6, see for
> example: https://issues.apache.org/jira/browse/BEAM-6877, which makes
> typehints support unusable on Python 3.6 as of now. [Python 3 Kanban
> Board|https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=245&view=detail]
> lists several specific typehints-related breakages, prefixed with "TypeHints
> Py3 Error".
> We need to decide whether to:
> - Deprecate in-house typehints implementation.
> - Continue to support in-house implementation, which at this point is a stale
> code and has other known issues.
> - Attempt to use some off-the-shelf libraries for supporting
> type-annotations, like Pytype, Mypy, PyAnnotate.
> WRT to this decision we also need to plan on immediate next steps to unblock
> adoption of Beam for Python 3.6+ users. One potential option may be to have
> Beam SDK ignore any typehint annotations on Py 3.6+.
> cc: [~udim], [~altay], [~robertwb].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
