I think that might be possible too. Yohei Onishi
On Tue, Jun 4, 2019 at 9:00 AM Chamikara Jayalath <[email protected]> wrote: > > > On Mon, Jun 3, 2019 at 5:14 PM Yohei Onishi <[email protected]> wrote: > >> Hi Nicolas, >> >> Are you running you job on Dataflow? According to GCP support, Dataflow >> currently does not support Schema Registry. But we can still use Schema >> Registry to serialize / deserialize your message using >> custom KafkaAvroSerializer. >> >> In my case I implemented my custom KafkaAvroDeserializer to deserialize >> Avro format message. >> You can implement custom KafkaAvroSerializer in a similar way. >> >> https://stackoverflow.com/questions/54755668/how-to-deserialising-kafka-avro-messages-using-apache-beam/55917157#55917157 >> >> I hope this helps you. >> >> Yohei Onishi >> >> >> On Wed, May 29, 2019 at 5:44 PM Nicolas Delsaux <[email protected]> >> wrote: >> >>> Hello all >>> >>> I have a beam job that I use to read messages from RabbitMq t write them >>> in kafka. >>> >>> As of now, messages are read/written as JSON. >>> >>> Obviously, it's not that optimal storage, so i would like to transform >>> the messages to avro prior to write them in Kafka. I have the URL of a >>> schema registry I can use to store/get my schema. >>> >> > Can you simply have a ParDo that sits in between read and write steps and > perform the conversion (and also connect to schema registry) ? > > >> >>> But I see nowhere in Beam documentation how to transform my JSON into >>> Avro data (except by deserializing my JSON to a java class that i will >>> later transform into avro). Is that deserialization to class the only >>> way ? or is it possible to generate an avro generic record from my json >>> "directly" ? >>> >>> Once my avro data is generated, how can I write it to my Kafka topic ? >>> >>> >>> Thanks ! >>> >>>
