Re: Error restoring Flink checkpoint

Could we try using schemas? On Thu, Jun 4, 2020 at 9:40 AM Maximilian Michels wrote: > I was under the assumption that this should work but ProtoCoder keeps a > reference of the class used to serialize. That causes the snapshot to > break. > > We can fix this by: > > a) writing/using coders

Re: Error restoring Flink checkpoint

I was under the assumption that this should work but ProtoCoder keeps a reference of the class used to serialize. That causes the snapshot to break. We can fix this by: a) writing/using coders which do not keep instances of evolving classes b) adding an interface to Beam for Coder

Re: Error restoring Flink checkpoint

I think Avro has two modes of operation. Either you supply a schema upfront or you let it auto-extract from a given type. I don't know how the auto-extraction works but if you specify the schema and version in your Beam Avro coder, then Avro will automatically resolve the checkpoint data against

Scio v0.9.1 release

Hi all, We just released scio v0.9.1. There are no breaking changes in this release, but some were introduced with v0.9.0, please our Migration Guide for detailed instructions. Cheers, Filipe Regadas

Re: Error restoring Flink checkpoint

I've changed my Java model in order to use ProtoCoder (with @DefaultCoder(ProtoCoder.class)), but I'm getting same error when tryingto restore the last taken checkpoint after adding an attribute to that model. What do you think it could happen? It seems that state saved within the checkpoint is

Re: Error restoring Flink checkpoint

Thanks Max for your response. I'd try with AvroCoder then. But I still have a question, I guess AvroCoder is generating the AVRO schema using Java reflection, and then that generated schema is saved within the Flink checkpoint, right? On Wed, 2020-06-03 at 18:00 +0200, Maximilian Michels wrote: >

EOFException when restarting BEAM job on Flink Runner

Hi all, I have an Apache BEAM pipeline failing to restart on the Flink Runner against a checkpoint. The exception is showing an EOF Exception related to the InstantCoder. This is version 2.20 of BEAM on 1.8 Flink. See stack trace below. This seems to be the same as the issue raised (against