[
https://issues.apache.org/jira/browse/BEAM-7060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16850078#comment-16850078
]
Chad Dombrova commented on BEAM-7060:
-------------------------------------
{quote}For example, a PTransform may be parameterized by an SQL statement where
the output type(s) would then depend on the input types(s) and the statement
itself. It's also hard to type things like hetrogenious dictionaries.
{quote}
In the case of a SQL statement, that will be opaque to just about any type
inference system, so the best you can do is provide a way to manually override
the output type of the transform at runtime.
mypy has a way to reason about heterogeneous dictionaries in the form of
TypedDict: [https://mypy.readthedocs.io/en/latest/more_types.html#typeddict].
{quote}This kind of thing forces us to do more typechecking at runtime (but,
fortunately, before the pipeline is actually launched, so in some sense we have
a phase that sits between the traditional static and runtime phases)
{quote}
I still have a lot to learn about how Beam does type checking, but it seems to
me that if you were to generate some typed python code during this phase,
containing the user-specified parameters (including manual overrides to certain
TypeVars to cover the SQL statement case), you could run static type checking
on this augmented pipeline code that would be as good (or almost as good?) as
dynamic/runtime checking. You have the same amount of information available,
you're just encoding it in a way that's compatible with mypy.
{quote}Do you know if it's possible to extract any of mypy's type-compatibility
validation or inference code as a library?
{quote}
For now, the CLI is their only supported entry-point (other than the plugin API
for extending analysis to custom/opaque objects), and I assume that stance
won't change, at least until the project has reached maturity. The reason
that's been given is that mypy is still under very active development and the
authors don't want the baggage of maintaining backward compatibility for
additional APIs. In my case, I tried and failed to lobby them to open up their
test runner (not even the internals of mypy itself) for use in projects outside
of mypy, and it was a hard "no".
So, while it might be _possible_ to use mypy as a python library, it would be
very fragile, and I doubt the developers would accept any PRs to make it easier
to do so. The other consideration is that mypy is supported in python 3.5+
only.
Here's the issue that I created: [https://github.com/python/mypy/issues/4366]
Comment from Guido:
{quote}I have to repeat what I've said before – the mypy package is *not* a
public API and we don't make any attempts at preserving compatibility across
releases. Please don't build applications or libraries depending on mypy
internals. And that includes the mypy.test subpackage. Please.
{quote}
> Design Py3-compatible typehints annotation support in Beam 3.
> -------------------------------------------------------------
>
> Key: BEAM-7060
> URL: https://issues.apache.org/jira/browse/BEAM-7060
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
> Reporter: Valentyn Tymofieiev
> Assignee: Udi Meiri
> Priority: Major
>
> Existing [Typehints implementaiton in
> Beam|[https://github.com/apache/beam/blob/master/sdks/python/apache_beam/typehints/
> ] heavily relies on internal details of CPython implementation, and some of
> the assumptions of this implementation broke as of Python 3.6, see for
> example: https://issues.apache.org/jira/browse/BEAM-6877, which makes
> typehints support unusable on Python 3.6 as of now. [Python 3 Kanban
> Board|https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=245&view=detail]
> lists several specific typehints-related breakages, prefixed with "TypeHints
> Py3 Error".
> We need to decide whether to:
> - Deprecate in-house typehints implementation.
> - Continue to support in-house implementation, which at this point is a stale
> code and has other known issues.
> - Attempt to use some off-the-shelf libraries for supporting
> type-annotations, like Pytype, Mypy, PyAnnotate.
> WRT to this decision we also need to plan on immediate next steps to unblock
> adoption of Beam for Python 3.6+ users. One potential option may be to have
> Beam SDK ignore any typehint annotations on Py 3.6+.
> cc: [~udim], [~altay], [~robertwb].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
