Love it. I've merged it, since it was already approved by Robert - and yes, we don't want to hit a merge conflict.
On Wed, Dec 11, 2019 at 1:36 PM Heejong Lee <heej...@google.com> wrote: > Wow, this is a big step forward. As a long fan of strongly typed > functional languages, I'm glad to see this change :) > > On Wed, Dec 11, 2019 at 9:44 AM Chad Dombrova <chad...@gmail.com> wrote: > >> Hi all, >> Robert has diligently reviewed the first batch of changes for this PR, >> and all review notes are addressed and tests are passing: >> https://github.com/apache/beam/pull/9915 >> >> Due to the number of file touched there's a short window of about one or >> two days before a merge conflict arrives on master, and after resolving >> that it usually takes another 1-2 days of pasting "Run Python PreCommit" >> until they pass again, so it would be great to get this merged while the >> window is open! Despite the number of files touched, the changes are >> almost entirely type comments, so the PR is designed to be quite safe. >> >> -chad >> >> >> On Tue, Nov 5, 2019 at 2:50 PM Chad Dombrova <chad...@gmail.com> wrote: >> >>> Glad to hear we have such a forward-thinking community! >>> >>> >>> On Tue, Nov 5, 2019 at 2:43 PM Robert Bradshaw <rober...@google.com> >>> wrote: >>> >>>> Sounds like we have consensus. Let's move forward. I'll follow up with >>>> the discussions on the PRs themselves. >>>> >>>> On Wed, Oct 30, 2019 at 2:38 PM Robert Bradshaw <rober...@google.com> >>>> wrote: >>>> > >>>> > On Wed, Oct 30, 2019 at 1:26 PM Chad Dombrova <chad...@gmail.com> >>>> wrote: >>>> > > >>>> > >> Do you believe that a future mypy plugin could replace pipeline >>>> type checks in Beam, or are there limits to what it can do? >>>> > > >>>> > > mypy will get us quite far on its own once we completely annotate >>>> the beam code. That said, my PR does not include my efforts to turn >>>> PTransforms into Generics, which will be required to properly analyze >>>> pipelines, so there's still a lot more work to do. I've experimented with >>>> a mypy plugin to smooth over some of the rough spots in that workflow and I >>>> will just say that the mypy API has a very steep learning curve. >>>> > > >>>> > > Another thing to note: mypy is very explicit about function >>>> annotations. It does not do the "implicit" inference that Beam does, such >>>> as automatically detecting function return types. I think it should be >>>> possible to do a lot of that as a mypy plugin, and in fact, since it has >>>> little to do with Beam it could grow into its own project with outside >>>> contributors. >>>> > >>>> > Yeah, I don't think, as is, it can replace what we do, but with >>>> > plugins I think it could possibly come closer. Certainly there is >>>> > information that is only available at runtime (e.g. reading from a >>>> > database or avro/parquet file could provide the schema which can be >>>> > used for downstream checking) which may limit the ability to do >>>> > everything statically (even Beam Java is moving this direction). Mypy >>>> > clearly has an implementation of the "is compatible with" operator >>>> > that I would love to borrow, but unfortunately it's not (easily?) >>>> > exposed. >>>> > >>>> > That being said, we should leverage what we can for pipeline >>>> > authoring, and it'll be a great development too regardless. >>>> >>>
