Thank you for response! > AFAIK, there's no official guide for cross-language pipelines. But there are > examples and test cases you can use as reference such as: > https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/wordcount_xlang.py > > <https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/wordcount_xlang.py> > https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubIOExternalTest.java > > <https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubIOExternalTest.java> > https://github.com/apache/beam/blob/master/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/ValidateRunnerXlangTest.java > > <https://github.com/apache/beam/blob/master/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/ValidateRunnerXlangTest.java> > https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/expansion_service_test.py > > <https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/expansion_service_test.py> >
More to this, are there any tests/examples showing how to execute an external Python transform (user code, not only IOs transforms) from Java pipeline? > > - Is this something that already can be used (currently interested in > Java/Python pipelines) or the main work is still in progress? More precisely > - I’m more focused in executing some Python code from Java-based pipelines. > > The runner and SDK supports are in working state I could say but not many IOs > expose their cross-language interface yet (you can easily write > cross-language configuration for any Python transforms by yourself though). If I understand that correctly, every runner should have a support for cross-language transforms additionally to portability support. If so, do you know if someone works on adding such support to Spark Runner?
