On Tue, Feb 11, 2020 at 9:37 AM Alexey Romanenko <[email protected]> wrote:
> Hi all, > > I just wanted to ask for more details about the status of cross-language > pipelines (rather, transforms). I see some discussions about that here, but > I think it’s more around cross-language IOs. > > I’ll appreciate for any information about that topic and answers for these > questions: > - Are there any examples/guides of setting up and running cross-languages > pipelines? > AFAIK, there's no official guide for cross-language pipelines. But there are examples and test cases you can use as reference such as: https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/wordcount_xlang.py https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubIOExternalTest.java https://github.com/apache/beam/blob/master/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/ValidateRunnerXlangTest.java https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/expansion_service_test.py > - Is this something that already can be used (currently interested in > Java/Python pipelines) or the main work is still in progress? More > precisely - I’m more focused in executing some Python code from Java-based > pipelines. > The runner and SDK supports are in working state I could say but not many IOs expose their cross-language interface yet (you can easily write cross-language configuration for any Python transforms by yourself though). > - Is the information here > https://beam.apache.org/roadmap/connectors-multi-sdk/ up-to-date? Are > there any other entry points you can recommend? > I think it's up-to-date. > > Thanks!
