Hi all, Robert has diligently reviewed the first batch of changes for this PR, and all review notes are addressed and tests are passing: https://github.com/apache/beam/pull/9915
Due to the number of file touched there's a short window of about one or two days before a merge conflict arrives on master, and after resolving that it usually takes another 1-2 days of pasting "Run Python PreCommit" until they pass again, so it would be great to get this merged while the window is open! Despite the number of files touched, the changes are almost entirely type comments, so the PR is designed to be quite safe. -chad On Tue, Nov 5, 2019 at 2:50 PM Chad Dombrova <chad...@gmail.com> wrote: > Glad to hear we have such a forward-thinking community! > > > On Tue, Nov 5, 2019 at 2:43 PM Robert Bradshaw <rober...@google.com> > wrote: > >> Sounds like we have consensus. Let's move forward. I'll follow up with >> the discussions on the PRs themselves. >> >> On Wed, Oct 30, 2019 at 2:38 PM Robert Bradshaw <rober...@google.com> >> wrote: >> > >> > On Wed, Oct 30, 2019 at 1:26 PM Chad Dombrova <chad...@gmail.com> >> wrote: >> > > >> > >> Do you believe that a future mypy plugin could replace pipeline type >> checks in Beam, or are there limits to what it can do? >> > > >> > > mypy will get us quite far on its own once we completely annotate the >> beam code. That said, my PR does not include my efforts to turn >> PTransforms into Generics, which will be required to properly analyze >> pipelines, so there's still a lot more work to do. I've experimented with >> a mypy plugin to smooth over some of the rough spots in that workflow and I >> will just say that the mypy API has a very steep learning curve. >> > > >> > > Another thing to note: mypy is very explicit about function >> annotations. It does not do the "implicit" inference that Beam does, such >> as automatically detecting function return types. I think it should be >> possible to do a lot of that as a mypy plugin, and in fact, since it has >> little to do with Beam it could grow into its own project with outside >> contributors. >> > >> > Yeah, I don't think, as is, it can replace what we do, but with >> > plugins I think it could possibly come closer. Certainly there is >> > information that is only available at runtime (e.g. reading from a >> > database or avro/parquet file could provide the schema which can be >> > used for downstream checking) which may limit the ability to do >> > everything statically (even Beam Java is moving this direction). Mypy >> > clearly has an implementation of the "is compatible with" operator >> > that I would love to borrow, but unfortunately it's not (easily?) >> > exposed. >> > >> > That being said, we should leverage what we can for pipeline >> > authoring, and it'll be a great development too regardless. >> >
