Yue Ma created FLINK-37551:
------------------------------

             Summary: Improve the state compatibility of ForStKeyeStateBackend 
and ForStSyncKeyeStateBackend
                 Key: FLINK-37551
                 URL: https://issues.apache.org/jira/browse/FLINK-37551
             Project: Flink
          Issue Type: Bug
          Components: Runtime / State Backends
    Affects Versions: 2.0.0
            Reporter: Yue Ma
             Fix For: 2.0.1


Currently, when switching between ForStKeyeStateBackend and 
ForStSyncKeyeStateBackend, the states may not be compatible. This may cause a 
lot of upgrade costs for users. The main purpose of this ticket is to improve 
the compatibility of ForStKeyeStateBackend and ForStSyncKeyeStateBackend.

In order to support the sync/async mode compatibility between 
ForStKeyedStateBackend, there may be many details involved. For the convenience 
of future tracking, the required changes will be summarized here. (The current 
list may still be incomplete, but it will be continuously updated until full 
compatibility)
 * {*}Made StateMetaInfo consistent for ForStKeyeStateBackend and 
ForStSyncKeyeStateBackend{*}.  Currently, if MapState or ListState is used, the 
StateMeta information saved by ForStKeyeStateBackend and 
ForStSyncKeyeStateBackend when making cp may be inconsistent, which may cause 
incompatible states. For example, when using MapState, ForStKeyeStateBackend 
will save the ValueSerializer information in StateMeta, but 
ForStSyncKeyeStateBackend will save the MapSerializer information. So we need 
to unify this place.
 * {*}Let ForStKeyedStateBackend consider namespaceSerializer compatibility 
when restoring state{*}. ForStKeyedStateBackend currently does not consider 
namespaceSerializer compatibility when restoring state. This may cause the 
namespaceSerializer to change when it is not known whether it should be 
compatible.
 * {*}Make StateMeta in ForState restore from oldStateInfo from  
Checkpoint{*}.(https://issues.apache.org/jira/browse/FLINK-37524) Currently, 
when creating using ForStKeyedStateBackend create a new State , 
AbstractKeyedState will directly use the serializer in StateDescriptor 
regardless of whether it is restored from a Checkpoint.However, this may cause 
data to be unable to be restored when the last serializer changes (for example, 
the class registration order in KryoSerializer has changed, for specific cases, 
please refer 
toStateBackendTestBase#testKryoRestoreResilienceWithDifferentRegistrationOrder).
 Therefore, when creating a State from a Checkpoint, oldStateInfo should be 
used as the new State's Serializer.
 *  *Add unit tests for state compatibility between ForStKeyeStateBackend and 
ForStSyncKeyeStateBackend*
 * *Make serialization consistent between ForStMapState and ForStSyncMapState* 
(https://issues.apache.org/jira/browse/FLINK-37419)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Reply via email to