Hi everyone,
The fact that core Beam has a dependency on Avro has led to a lot of
headaches when users (or runners) are using a different version. zeidoo [1]
was generous enough to put up a WIP PR [2] that moves everything that
depends on Avro (primarily AvroCoder and the Avro SchemaProvider I believe)
out of core Beam and into a separate extensions module. This way we could
have multiple extensions for different versions of Avro in the future.

As I understand it, the downsides to making this change are:
1) It's a breaking change, users with AvroCoder in their pipeline will need
to change their build dependencies and import statements.
2) AvroCoder is the only (non-experimental) coder in core Beam that can
encode complex user types. So new users will need to dabble with the
Experimental SchemaCoder or add a second dependency to build a pipeline
with their own types.

I think these costs are outweighed by the benefit of removing the
dependency in core Beam, but I wanted to reach out to the community to see
if there are any objections.

Brian

[1] github.com/zeidoo
[2] https://github.com/apache/beam/pull/12748

Reply via email to