+cc dev@beam Hello Daisy! There presently isn't a Kafka transform for Go. The Go SDK is still experimental, largely due to scalable IO support, which is why the Go SDK isn't represented in the built-in io page <https://beam.apache.org/documentation/io/built-in/>.
There's presently no way for an SDK user to write a Streaming source in the Go SDK, since there's no mechanism for a DoFn to "self terminate" bundles, such as to allow for scalability and windowing from streaming sources. However, SplittableDoFns are on their way, and will eventually be the solution for writing these. At present, the Beam Go SDK IOs haven't been tested and vetted for production use. Until the initial SplittableDoFn support is added to the Go SDK, Batch transforms cannot split, and can't scale beyond a single worker thread. This batch version should land in the next few months, and the streaming version land a few months after that, after which a Kafka IO can be developed. I wish I had better news for you, but I can say progress is being made. Robert Burke On Sun, Jan 26, 2020 at 10:14 PM Daisy Wu <[email protected]> wrote: > Hi, Robert, > > I found your name from the Apache Beam WIKI page. > > I am working on building a data ingestion pipeline using Apache Beam "go" > SDK. > > My pipeline is to consume data from Kafka queue and persist the data to > Google Cloud Bigtable (and/or to another Kafka topic). > > So far, I have not been able to find a Kafka IO Connector (also known as > Apache I/O Transform) written in "go" (I was able to find a java version, > however). > > Here's link to supported Apache Beam built-in I/O transforms: > https://beam.apache.org/documentation/io/built-in/ > > I am looking for the "go" equivalent of the following Java code: > > pipeline.apply("kafka_deserialization", KafkaIO.<String, String>read() > .withBootstrapServers(KAFKA_BROKER) > .withTopic(KAFKA_TOPIC) > .withConsumerConfigUpdates(CONSUMER_CONFIG) > .withKeyDeserializer(StringDeserializer.class) > .withValueDeserializer(StringDeserializer.class)) > > Do you have any information on the availability of KafkaIO > Connector/Transform "go" SDK/library? > > Any help or information would be much appreciated. > > Thank you. > > > Daisy Wu >
