Currently the Go SDK doesn't have cross Language support implemented. My recommendation is that we make sure the protocol is stable and implemented on the Python SDK side before starting the Go SDK side, since that work is already in progress.
The relevant state of the Go SDK: * beam.External exists, for specifying go transforms. (See the Go SDK's PubSubIO for an example) * the generated go code for the protos, including the Expansions service API was refreshed last week. Meaning the work isn't blocked on that. In principle, the work would be to * ensure that the SDK side of job submission connects and looks up relevant transforms against the Expansion service if needed, and does the appropriate pipeline graph surgery. *This may be something that's handled as some kind of hook and registration process for generally handling external transforms SDK side. * Have some kind of external transform to specify and configure on the Go side. Most of this can be ported from the Python implementation once it's stabilized. As with all my recommendations on how to do things with the Go SDK, feel free to ignore it and forge ahead. I look forward to someone tackling this, whenever it happens! Your friendly neighborhood distributed gopher wrangler, Robert Burke Related: PR 8531 [1] begins adding automates testing of the Go SDK against Flink, which should assist with ensuring this eventual work keeps working. [1]: https://github.com/apache/beam/pull/8531 On Thu, May 9, 2019, 6:32 AM Łukasz Gajowy <lgaj...@apache.org> wrote: > Hi, > > part of our work that needs to be done to create tests for Core Apache > Beam operations is to enable both batch and streaming testing scenarios in > all SDKs (not only Java, so lot's of portability usage here). I gathered > some thoughts on how (I think) this could be done at the current state of > Beam: > > https://s.apache.org/portable-load-tests > > I added some questions at the end of the doc but I will paste them here > too for visibility: > > - What is the status of Cross Language Support for Go SDK? Is it > non-trivial to enable such support (if it doesn't exist yet)? > - According to other contributors knowledge/experience: I noticed that > streaming with KafkaIO is currently supported by wrapping the > ExternalTransform in Python SDK. Do you think that streaming pipelines will > "just work" with the current state of portability if I do the same for > UnboundedSyntheticSource or is there something else missing? > > BTW: great to see Cross Language Support happening. Thanks for doing this! > :) > > Thanks, > Łukasz > > > > >
