Great! Thanks for your help as well! Happy that my change made it through to the master.
On Mon, Mar 18, 2024 at 7:34 PM Yi Hu <ya...@google.com> wrote: > +1 to move forward with avro. Users can still stick with old avro version > if chosen to do so, and we have Avro compatibility tests also passed in > https://github.com/apache/beam/pull/30638 > > Thanks for doing this! > > On Fri, Mar 15, 2024 at 3:30 PM Maciej Szwaja via dev <dev@beam.apache.org> > wrote: > >> Hi team, thanks for the answers and an update from me - I've created the >> PR https://github.com/apache/beam/pull/30638 which seems to pass the >> tests (sometimes they're flaky though, so I didn't manage to get all green >> in the checks yet), and also I was able to run a pipeline that reads avro >> records from kafka topic and then writes them down to bigquery. >> >> In the meantime though, Yi Hu pointed out that updating the avro version >> (and I need to do that if I want to update the confluent libs to the most >> recent version) is not a trivial issue, and there's been discussions around >> it in the past. I must admit I'd probably have to study the beam codebase a >> bit more to understand the risks around it - so I wanted to ask for advice >> on how to proceed. I could try updating the confluent version more >> gradually - but I think even the 5.4.0, which is the lowest one that >> includes the functionality that I wanted to have, would still probably >> require upgrading to avro 1.9.2. On the other hand - maybe the reservations >> that were relevant earlier (for example - the leaking of avro lib to the >> users' code as a result of it being part of the java sdk core - I think >> it's been migrated out to its own extension since then if I'm not mistaken) >> are no longer that worrisome, and we could try to finally move forward with >> avro? >> >
