[
https://issues.apache.org/jira/browse/FLINK-37551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yue Ma updated FLINK-37551:
---------------------------
Description:
Currently, when switching between ForStKeyeStateBackend and
ForStSyncKeyeStateBackend, the states may not be compatible. This may cause a
lot of upgrade costs for users. The main purpose of this ticket is to improve
the compatibility of ForStKeyeStateBackend and ForStSyncKeyeStateBackend.
In order to support the sync/async mode compatibility between
ForStKeyedStateBackend, there may be many details involved. For the convenience
of future tracking, the required changes will be summarized here. (The current
list may still be incomplete, but it will be continuously updated until full
compatibility)
* {*}Made StateMetaInfo consistent for ForStKeyeStateBackend and
ForStSyncKeyeStateBackend{*}. Currently, if MapState or ListState is used, the
StateMeta information saved by ForStKeyeStateBackend and
ForStSyncKeyeStateBackend when making cp may be inconsistent, which may cause
incompatible states. For example, when using MapState, ForStKeyeStateBackend
will save the ValueSerializer information in StateMeta, but
ForStSyncKeyeStateBackend will save the MapSerializer information. So we need
to unify this place and perform a transformation when ForStKeyeStateBackend
creates stateMeta
* {*}Make ForStKeyedStateBackend consider namespaceSerializer compatibility
when restoring state{*}. ForStKeyedStateBackend currently does not consider
namespaceSerializer compatibility when restoring state. This may cause the
namespaceSerializer to change when it is not known whether it should be
compatible.
* {*}Make StateMeta in ForState restore from oldStateInfo from
Checkpoint{*}.(https://issues.apache.org/jira/browse/FLINK-37524) Currently,
when creating using ForStKeyedStateBackend create a new State ,
AbstractKeyedState will directly use the serializer in StateDescriptor
regardless of whether it is restored from a Checkpoint.However, this may cause
data to be unable to be restored when the last serializer changes (for example,
the class registration order in KryoSerializer has changed, for specific cases,
please refer
toStateBackendTestBase#testKryoRestoreResilienceWithDifferentRegistrationOrder).
Therefore, when creating a State from a Checkpoint, oldStateInfo should be
used as the new State's Serializer.
* *Add unit tests for state compatibility between ForStKeyeStateBackend and
ForStSyncKeyeStateBackend*
* *Make serialization consistent between ForStMapState and ForStSyncMapState*
(https://issues.apache.org/jira/browse/FLINK-37419)
was:
Currently, when switching between ForStKeyeStateBackend and
ForStSyncKeyeStateBackend, the states may not be compatible. This may cause a
lot of upgrade costs for users. The main purpose of this ticket is to improve
the compatibility of ForStKeyeStateBackend and ForStSyncKeyeStateBackend.
In order to support the sync/async mode compatibility between
ForStKeyedStateBackend, there may be many details involved. For the convenience
of future tracking, the required changes will be summarized here. (The current
list may still be incomplete, but it will be continuously updated until full
compatibility)
* {*}Made StateMetaInfo consistent for ForStKeyeStateBackend and
ForStSyncKeyeStateBackend{*}. Currently, if MapState or ListState is used, the
StateMeta information saved by ForStKeyeStateBackend and
ForStSyncKeyeStateBackend when making cp may be inconsistent, which may cause
incompatible states. For example, when using MapState, ForStKeyeStateBackend
will save the ValueSerializer information in StateMeta, but
ForStSyncKeyeStateBackend will save the MapSerializer information. So we need
to unify this place and perform a transformation when ForStKeyeStateBackend
creates stateMeta
* {*}Let ForStKeyedStateBackend consider namespaceSerializer compatibility
when restoring state{*}. ForStKeyedStateBackend currently does not consider
namespaceSerializer compatibility when restoring state. This may cause the
namespaceSerializer to change when it is not known whether it should be
compatible.
* {*}Make StateMeta in ForState restore from oldStateInfo from
Checkpoint{*}.(https://issues.apache.org/jira/browse/FLINK-37524) Currently,
when creating using ForStKeyedStateBackend create a new State ,
AbstractKeyedState will directly use the serializer in StateDescriptor
regardless of whether it is restored from a Checkpoint.However, this may cause
data to be unable to be restored when the last serializer changes (for example,
the class registration order in KryoSerializer has changed, for specific cases,
please refer
toStateBackendTestBase#testKryoRestoreResilienceWithDifferentRegistrationOrder).
Therefore, when creating a State from a Checkpoint, oldStateInfo should be
used as the new State's Serializer.
* *Add unit tests for state compatibility between ForStKeyeStateBackend and
ForStSyncKeyeStateBackend*
* *Make serialization consistent between ForStMapState and ForStSyncMapState*
(https://issues.apache.org/jira/browse/FLINK-37419)
> Improve the state compatibility of ForStKeyeStateBackend and
> ForStSyncKeyeStateBackend
> --------------------------------------------------------------------------------------
>
> Key: FLINK-37551
> URL: https://issues.apache.org/jira/browse/FLINK-37551
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / State Backends
> Affects Versions: 2.0.0
> Reporter: Yue Ma
> Assignee: Yue Ma
> Priority: Major
> Fix For: 2.1.0
>
>
> Currently, when switching between ForStKeyeStateBackend and
> ForStSyncKeyeStateBackend, the states may not be compatible. This may cause a
> lot of upgrade costs for users. The main purpose of this ticket is to improve
> the compatibility of ForStKeyeStateBackend and ForStSyncKeyeStateBackend.
> In order to support the sync/async mode compatibility between
> ForStKeyedStateBackend, there may be many details involved. For the
> convenience of future tracking, the required changes will be summarized here.
> (The current list may still be incomplete, but it will be continuously
> updated until full compatibility)
> * {*}Made StateMetaInfo consistent for ForStKeyeStateBackend and
> ForStSyncKeyeStateBackend{*}. Currently, if MapState or ListState is used,
> the StateMeta information saved by ForStKeyeStateBackend and
> ForStSyncKeyeStateBackend when making cp may be inconsistent, which may cause
> incompatible states. For example, when using MapState, ForStKeyeStateBackend
> will save the ValueSerializer information in StateMeta, but
> ForStSyncKeyeStateBackend will save the MapSerializer information. So we need
> to unify this place and perform a transformation when ForStKeyeStateBackend
> creates stateMeta
> * {*}Make ForStKeyedStateBackend consider namespaceSerializer compatibility
> when restoring state{*}. ForStKeyedStateBackend currently does not consider
> namespaceSerializer compatibility when restoring state. This may cause the
> namespaceSerializer to change when it is not known whether it should be
> compatible.
> * {*}Make StateMeta in ForState restore from oldStateInfo from
> Checkpoint{*}.(https://issues.apache.org/jira/browse/FLINK-37524) Currently,
> when creating using ForStKeyedStateBackend create a new State ,
> AbstractKeyedState will directly use the serializer in StateDescriptor
> regardless of whether it is restored from a Checkpoint.However, this may
> cause data to be unable to be restored when the last serializer changes (for
> example, the class registration order in KryoSerializer has changed, for
> specific cases, please refer
> toStateBackendTestBase#testKryoRestoreResilienceWithDifferentRegistrationOrder).
> Therefore, when creating a State from a Checkpoint, oldStateInfo should be
> used as the new State's Serializer.
> * *Add unit tests for state compatibility between ForStKeyeStateBackend and
> ForStSyncKeyeStateBackend*
> * *Make serialization consistent between ForStMapState and
> ForStSyncMapState* (https://issues.apache.org/jira/browse/FLINK-37419)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
