On Tue, Feb 11, 2020 at 11:13 AM Heejong Lee <[email protected]> wrote:
> > > On Tue, Feb 11, 2020 at 9:37 AM Alexey Romanenko <[email protected]> > wrote: > >> Hi all, >> >> I just wanted to ask for more details about the status of cross-language >> pipelines (rather, transforms). I see some discussions about that here, but >> I think it’s more around cross-language IOs. >> >> I’ll appreciate for any information about that topic and answers for >> these questions: >> - Are there any examples/guides of setting up and running cross-languages >> pipelines? >> > > AFAIK, there's no official guide for cross-language pipelines. But there > are examples and test cases you can use as reference such as: > > https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/wordcount_xlang.py > > https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubIOExternalTest.java > > https://github.com/apache/beam/blob/master/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/ValidateRunnerXlangTest.java > > https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/expansion_service_test.py > I'm trying to work with tech writers to add more documentation related to cross-language (in a few months). But any help related to documenting what we have now is greatly appreciated. > > > > >> - Is this something that already can be used (currently interested in >> Java/Python pipelines) or the main work is still in progress? More >> precisely - I’m more focused in executing some Python code from Java-based >> pipelines. >> > > The runner and SDK supports are in working state I could say but not many > IOs expose their cross-language interface yet (you can easily write > cross-language configuration for any Python transforms by yourself though). > Should mention here the test suites for portable Flink and Spark Heejong added recently :) https://builds.apache.org/view/A-D/view/Beam/view/PostCommit/job/beam_PostCommit_XVR_Flink/ https://builds.apache.org/view/A-D/view/Beam/view/PostCommit/job/beam_PostCommit_XVR_Spark/ > > >> - Is the information here >> https://beam.apache.org/roadmap/connectors-multi-sdk/ up-to-date? Are >> there any other entry points you can recommend? >> > > I think it's up-to-date. > Mostly up to date. Testing status is more complete now and we are actively working on getting the dependences story correct and adding support for DataflowRunner. > > >> >> Thanks! > >
