[
https://issues.apache.org/jira/browse/FLINK-6425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tzu-Li (Gordon) Tai updated FLINK-6425:
---------------------------------------
Description:
With FLINK-6191, {{TypeSerializer}} will be reconfigurable.
>From the state backends' point of view, serializer reconfiguration doubles as
>a mechanism to determine how serializer upgrades should be handled.
The general idea is that state checkpoints should contain the following as the
state's metainfo:
- the previous serializer
- snapshot of the previous serializer's configuration
The upgrade flow is as follows:
1. On restore, try to deserialize the previous old serializer. Deserialization
may fail if a) the serializer no longer exists in classpath, or b) the
serializer class is not longer valid (i.e., implementation changed and resulted
in different UUID). In this case, use a dummy serializer as a placeholder. This
dummy serializer is currently the {{ClassNotFoundProxySerializer}} in the code.
2. Deserialize the configuration snapshot of the previous old serializer. The
configuration snapshot must be successfully deserialized, otherwise the state
restore fails.
3. When we get the new registered serializer for the state (could be a
completely new serializer, the same serializer with different implementations,
or the exact same serializer untouched; either way they are seen as a new
serializer), we use the configuration snapshot of the old serializer to
reconfigure the new serializer.
This completes the upgrade of the old serializer. However, depending on the
result of the upgrade, state conversion needs to take place (for now, if state
conversion is required, we just fail the job as this functionality isn't
available yet). The results could be:
- Compatible: restore success + serializer upgraded.
- Compatible, but serialization schema changed: serializer upgraded but
requires state conversion, without the requirement that the old serializer
needs to be present.
- Incompatible: serializer upgraded requires state conversion, but requires the
old serializer to be present (i.e., can not be the dummy
{{ClassNotFoundProxySerializer}}).
was:
With FLINK-6191, {{TypeSerializer}} will be reconfigurable.
>From the state backends' point of view, serializer reconfiguration doubles as
>a mechanism to determine how serializer upgrades should be handled.
The general idea is that state checkpoints should contain the following as the
state's metainfo:
- the previous serializer
- snapshot of the previous serializer's configuration
The upgrade flow is as follows:
1. On restore, try to deserialize the previous old serializer. Deserialization
may fail if a) the serializer no longer exists in classpath, or b) the
serializer class is not longer valid (i.e., implementation changed and resulted
in different UUID). In this case, use a dummy serializer as a placeholder. This
dummy serializer is currently the {{ClassNotFoundProxySerializer}} in the code.
2. Deserialize the configuration snapshot of the previous old serializer. The
configuration snapshot must be successfully deserialized, otherwise the state
restore fails.
3. When we get the new registered serializer for the state (could be a
completely new serializer, the same serializer with different implementations,
or the exact same serializer untouched; either way they are seen as a new
serializer), we use the configuration snapshot of the old serializer to
reconfigure the new serializer.
This completes the upgrade of the old serializer. However, depending on the
result of the upgrade, state conversion needs to take place (for now, if state
conversion is required, we just fail the job as this functionality isn't
available yet). The results could be:
- Compatible: restore success + serializer upgraded.
- Compatible, but serialization schema changed: requires state conversion,
without the requirement that the old serializer needs to be present.
- Incompatible: requires state conversion, but requires the old serializer to
be present (i.e., can not be the dummy {{ClassNotFoundProxySerializer}}).
> Integrate serializer reconfiguration into state restore flow to activate
> serializer upgrades
> --------------------------------------------------------------------------------------------
>
> Key: FLINK-6425
> URL: https://issues.apache.org/jira/browse/FLINK-6425
> Project: Flink
> Issue Type: Sub-task
> Components: State Backends, Checkpointing
> Reporter: Tzu-Li (Gordon) Tai
> Assignee: Tzu-Li (Gordon) Tai
>
> With FLINK-6191, {{TypeSerializer}} will be reconfigurable.
> From the state backends' point of view, serializer reconfiguration doubles as
> a mechanism to determine how serializer upgrades should be handled.
> The general idea is that state checkpoints should contain the following as
> the state's metainfo:
> - the previous serializer
> - snapshot of the previous serializer's configuration
> The upgrade flow is as follows:
> 1. On restore, try to deserialize the previous old serializer.
> Deserialization may fail if a) the serializer no longer exists in classpath,
> or b) the serializer class is not longer valid (i.e., implementation changed
> and resulted in different UUID). In this case, use a dummy serializer as a
> placeholder. This dummy serializer is currently the
> {{ClassNotFoundProxySerializer}} in the code.
> 2. Deserialize the configuration snapshot of the previous old serializer. The
> configuration snapshot must be successfully deserialized, otherwise the state
> restore fails.
> 3. When we get the new registered serializer for the state (could be a
> completely new serializer, the same serializer with different
> implementations, or the exact same serializer untouched; either way they are
> seen as a new serializer), we use the configuration snapshot of the old
> serializer to reconfigure the new serializer.
> This completes the upgrade of the old serializer. However, depending on the
> result of the upgrade, state conversion needs to take place (for now, if
> state conversion is required, we just fail the job as this functionality
> isn't available yet). The results could be:
> - Compatible: restore success + serializer upgraded.
> - Compatible, but serialization schema changed: serializer upgraded but
> requires state conversion, without the requirement that the old serializer
> needs to be present.
> - Incompatible: serializer upgraded requires state conversion, but requires
> the old serializer to be present (i.e., can not be the dummy
> {{ClassNotFoundProxySerializer}}).
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
