+1 to move forward with avro. Users can still stick with old avro version if chosen to do so, and we have Avro compatibility tests also passed in https://github.com/apache/beam/pull/30638
Thanks for doing this! On Fri, Mar 15, 2024 at 3:30 PM Maciej Szwaja via dev <dev@beam.apache.org> wrote: > Hi team, thanks for the answers and an update from me - I've created the > PR https://github.com/apache/beam/pull/30638 which seems to pass the > tests (sometimes they're flaky though, so I didn't manage to get all green > in the checks yet), and also I was able to run a pipeline that reads avro > records from kafka topic and then writes them down to bigquery. > > In the meantime though, Yi Hu pointed out that updating the avro version > (and I need to do that if I want to update the confluent libs to the most > recent version) is not a trivial issue, and there's been discussions around > it in the past. I must admit I'd probably have to study the beam codebase a > bit more to understand the risks around it - so I wanted to ask for advice > on how to proceed. I could try updating the confluent version more > gradually - but I think even the 5.4.0, which is the lowest one that > includes the functionality that I wanted to have, would still probably > require upgrading to avro 1.9.2. On the other hand - maybe the reservations > that were relevant earlier (for example - the leaking of avro lib to the > users' code as a result of it being part of the java sdk core - I think > it's been migrated out to its own extension since then if I'm not mistaken) > are no longer that worrisome, and we could try to finally move forward with > avro? >
