Tzu-Li (Gordon) Tai created FLINK-9808:

             Summary: Implement state conversion procedure in state backends
                 Key: FLINK-9808
             Project: Flink
          Issue Type: Sub-task
          Components: State Backends, Checkpointing
            Reporter: Tzu-Li (Gordon) Tai
            Assignee: Aljoscha Krettek
             Fix For: 1.6.0

With FLINK-9377 in place and that config snapshots serve as the single source 
of truth for recreating restore serializers, the next step would be to utilize 
this when performing a full-pass state conversion (i.e., read with old / 
restore serializer, write with new serializer).

For Flink's heap-based backends, it can be seen that state conversion 
inherently happens, since all state is always deserialized after restore with 
the restore serializer, and written with the new serializer on snapshots.

For the RocksDB state backend, since state is lazily deserialized, state 
conversion needs to happen for per-registered state on their first access if 
the registered new serializer has a different serialization schema than the 
previous serializer.

This task should consist of three parts:

1. Allow {{CompatibilityResult}} to correctly distinguish between whether the 
new serializer's schema is a) compatible with the serializer as it is, b) 
compatible after the serializer has been reconfigured, or c) incompatible.

2. Introduce state conversion procedures in the RocksDB state backend. This 
should occur on the first state access.

3. Make sure that all other backends no longer do redundant serializer 
compatibility checks. That is not required because those backends always 
perform full-pass state conversions.

This message was sent by Atlassian JIRA

Reply via email to