Re: Dataflow updates fail with "Coder has changed" error using KafkaIO with SchemaCoder

2020-10-14 Thread Cameron Morgan
We are using an Avro Schema Registry and converting these schemas to Beam Schemas with `AvroUtils.toBeamSchema`: https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/AvroUtils.java#L314

Re: Dataflow updates fail with "Coder has changed" error using KafkaIO with SchemaCoder

2020-10-09 Thread Brian Hulette
Hi Cameron, Thanks for bringing this up on the dev list. I'm quite familiar with Beam schemas, but I should be clear I'm not that familiar with Dataflow's pipeline update. +Reuven Lax may need to check me there. > I am curious if it has been determined what makes a Schema the same as another

Dataflow updates fail with "Coder has changed" error using KafkaIO with SchemaCoder

2020-10-08 Thread Cameron Morgan
Hey everyone, Summary: There is an issue with the Dataflow runner and the “Update” capability while using the beam native Row type, which I imagine also blocks the snapshots feature (as the docs say the snapshots have the same restrictions as the Update feature) but I have no experience