Stefan Richter created FLINK-5051:
-------------------------------------
Summary: Backwards compatibility for serializers in backend state
Key: FLINK-5051
URL: https://issues.apache.org/jira/browse/FLINK-5051
Project: Flink
Issue Type: Improvement
Components: State Backends, Checkpointing
Reporter: Stefan Richter
When a new state is register, e.g. in a keyed backend via
`getPartitionedState`, the caller has to provide all type serializers required
for the persistence of state components. Explicitly passing the serializers on
state creation already allows for potentiall version upgrades of serializers.
However, those serializers are currently not part of any snapshot and are only
provided at runtime, when the state is registered newly or restored. For
backwards compatibility, this has strong implications: checkpoints are not self
contained in that state is currently a blackbox without knowledge about it's
corresponding serializers. Most cases where we would need to restructure the
state are basically lost. We could only convert them lazily at runtime and only
once the user is registering the concrete state, which might happen at
unpredictable points.
I suggest to adapt our solution as follows:
- As now, all states are registered with their set of serializers.
- Unlike now, all serializers are written to the snapshot. This makes
savepoints self-contained and also allows to create inspection tools for
savepoints at some point in the future.
- Introduce an interface {{Versioned}} with {{long getVersion()}} and {{boolean
isCompatible(Versioned v)}} which is then implemented by serializers.
Compatible serializers must ensure that they can deserialize older versions,
and can then serialize them in their new format. This is how we upgrade.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
