Hi Nicolas, Are you running you job on Dataflow? According to GCP support, Dataflow currently does not support Schema Registry. But we can still use Schema Registry to serialize / deserialize your message using custom KafkaAvroSerializer.
In my case I implemented my custom KafkaAvroDeserializer to deserialize Avro format message. You can implement custom KafkaAvroSerializer in a similar way. https://stackoverflow.com/questions/54755668/how-to-deserialising-kafka-avro-messages-using-apache-beam/55917157#55917157 I hope this helps you. Yohei Onishi On Wed, May 29, 2019 at 5:44 PM Nicolas Delsaux <[email protected]> wrote: > Hello all > > I have a beam job that I use to read messages from RabbitMq t write them > in kafka. > > As of now, messages are read/written as JSON. > > Obviously, it's not that optimal storage, so i would like to transform > the messages to avro prior to write them in Kafka. I have the URL of a > schema registry I can use to store/get my schema. > > But I see nowhere in Beam documentation how to transform my JSON into > Avro data (except by deserializing my JSON to a java class that i will > later transform into avro). Is that deserialization to class the only > way ? or is it possible to generate an avro generic record from my json > "directly" ? > > Once my avro data is generated, how can I write it to my Kafka topic ? > > > Thanks ! > >
