Great job, it should be very helpful for users! Just a minor note - it would be great to add an example of how to finally run a cross-language pipeline with Portable Runner since, iirc, it was supposed to pass some additional arguments, like “--experiments=beam_fn_api”.
> On 21 Nov 2020, at 03:36, Chamikara Jayalath <chamik...@google.com> wrote: > > PR went in and documentation is live now: > https://beam.apache.org/documentation/programming-guide/#mulit-language-pipelines > > <https://beam.apache.org/documentation/programming-guide/#mulit-language-pipelines> > > Thanks, > Cham > > On Wed, Nov 18, 2020 at 10:05 AM Chamikara Jayalath <chamik...@google.com > <mailto:chamik...@google.com>> wrote: > This was mentioned in a separate thread but thought it would be good to > highlight here in case more folks wish to take a look before the PR is merged. > > PR is https://github.com/apache/beam/pull/13317 > <https://github.com/apache/beam/pull/13317> > > Thanks, > Cham > > On Thu, Nov 12, 2020 at 1:17 PM Chamikara Jayalath <chamik...@google.com > <mailto:chamik...@google.com>> wrote: > Seems like a good place to promote this PR that adds documentation for > cross-language transforms :) > https://github.com/apache/beam/pull/13317 > <https://github.com/apache/beam/pull/13317> > > This covers the following for both Java and Python SDKs. > * Creating new cross-language transforms - primary audience will be transform > authors who wish to make existing Java/Python transforms available to other > SDKs. > * Using cross-language transforms - primary audience will be pipeline authors > that wish to use existing cross-language transforms with or without language > specific wrappers. > > Also this introduces the term "Multi-Language Pipelines" to denote pipelines > that use cross-language transforms (and hence utilize more than one SDK > language). > > Thanks +Dave Wrede <mailto:dwr...@google.com> for working on this. > > - Cham > > On Thu, Nov 12, 2020 at 4:56 AM Ismaël Mejía <ieme...@gmail.com > <mailto:ieme...@gmail.com>> wrote: > I was not aware of these examples Brian, thanks for sharing. Maybe we should > make these examples more discoverable on the website or as part of Beam's > programming guide. > > It would be nice to have an example of the opposite too, calling a Python > transform from Java. > > Additionally Java users who want to integrate python might be lost because > External is NOT part of Beam's Java SDK (the transform is hidden inside of a > different module core-construction-java), so it does not even appear in the > website SDK javadoc. > https://issues.apache.org/jira/browse/BEAM-8546 > <https://issues.apache.org/jira/browse/BEAM-8546> > > > On Wed, Nov 11, 2020 at 8:41 PM Brian Hulette <bhule...@google.com > <mailto:bhule...@google.com>> wrote: > > > > Hi Ke, > > > > A cross-language pipeline looks a lot like a pipeline written natively in > > one of the Beam SDKs, the difference is that some of the transforms in the > > pipeline may be "external transforms" that actually have implementations in > > a different language. There are a few examples in the beam repo that use > > Java transforms from Python pipelines: > > - kafkataxi [1]: Uses Java's KafkaIO from Python > > - wordcount_xlang_sql [2] and sql_taxi [3]: Use Java's SqlTransform from > > Python > > > > To create your own cross-language pipeline, you'll need to decide which SDK > > you want to use primarily, and then create an expansion service to expose > > the transforms you want to use from the other SDK (if one doesn't exist > > already). > > > > [1] > > https://github.com/apache/beam/tree/master/sdks/python/apache_beam/examples/kafkataxi > > > > <https://github.com/apache/beam/tree/master/sdks/python/apache_beam/examples/kafkataxi> > > [2] > > https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/wordcount_xlang_sql.py > > > > <https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/wordcount_xlang_sql.py> > > [3] > > https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/sql_taxi.py > > > > <https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/sql_taxi.py> > > > > On Wed, Nov 11, 2020 at 11:07 AM Ke Wu <ke.wu...@gmail.com > > <mailto:ke.wu...@gmail.com>> wrote: > >> > >> Hello, > >> > >> Is there an example demonstrating how a cross language pipeline look like? > >> e.g. a pipeline where it is composes of Java and Python code/transforms. > >> > >> Best, > >> Ke
