Hi team, thanks for the answers and an update from me - I've created the PR https://github.com/apache/beam/pull/30638 which seems to pass the tests (sometimes they're flaky though, so I didn't manage to get all green in the checks yet), and also I was able to run a pipeline that reads avro records from kafka topic and then writes them down to bigquery.
In the meantime though, Yi Hu pointed out that updating the avro version (and I need to do that if I want to update the confluent libs to the most recent version) is not a trivial issue, and there's been discussions around it in the past. I must admit I'd probably have to study the beam codebase a bit more to understand the risks around it - so I wanted to ask for advice on how to proceed. I could try updating the confluent version more gradually - but I think even the 5.4.0, which is the lowest one that includes the functionality that I wanted to have, would still probably require upgrading to avro 1.9.2. On the other hand - maybe the reservations that were relevant earlier (for example - the leaking of avro lib to the users' code as a result of it being part of the java sdk core - I think it's been migrated out to its own extension since then if I'm not mistaken) are no longer that worrisome, and we could try to finally move forward with avro?
