> 1. The article in question probably makes use of Flink serialization, what
> if I use Avro serde for the serialization and deserialization part. If I
> create a savepoint of my job, stop my flink, load the new POJO and continue
> from the savepoint, would avro's schema evolution feature perform the
> transition smoothly? 
> For example, a new entity is inserted, all the old values would get a
> default value for which there is no value available and when an entity is
> deleted, that value is simply dropped?

Serializers can provide their own schema evolution. In this case, when the 
schema changes, the serializer would simply signal compatibility and deal with 
the schema versioning internally and transparently for Flink. However, the 
serializer should be able to deal with all schema versions at all time, because 
right now, e.g. for the RocksDB backend, it is impossible to tell if and when a 
state is updated and rewritten in the new schema because an explicit conversion 
step is currently still lacking (as described in the documentation). How such a 
serializer deals with new and dropped entities is up to the implementation, 
Flink will simply accept whatever the serializer delivers. So Avro schema 
evolution should work.

> 2. If yes, how would this play out in the flink ecosystem, and if not, would
> the flink serialization upgrades in the future handle such cases(forward and
> backward compatibility)?

As I see it, you can now use serializers that have their own schema evolution 
and Flink will probably offer an additional, explicit way of schema evolution.

> 3. Are managed state also stored and reloaded, when savepoints are created
> and used for resuming a job?

This depends on the definition of „stored/reloaded“ and on the state backend. 
If stored/reloaded means a roundtrip through serde, then the answer might be no 
for the RocksDB backend. This backend always contains the state serialized as 
bytes and goes through serde per access/update. The checkpoint/savepoint is 
based on the stored bytes. In contrast to that, heap based backends will go 
through a serialization on checkpoint/savepoint and deserialization on 

> 4. When can one expect to have the state migration feature in Flink? In
> 1.4.0? 

IIRC this is not part of the 1.4 roadmap. Flink 1.5 might be more realistic.


Reply via email to