tzulitai opened a new pull request #7422: [FLINK-11073] (part 2) Introduce CompositeTypeSerializerSnapshot and migrate existing composite serializers' snapshots URL: https://github.com/apache/flink/pull/7422 ## What is the purpose of the change This PR is based on top of #7329. Only the commits starting from 483dabb are relevant. #7329 introduced the new `COMPATIBLE_WITH_RECONFIGURED_SERIALIZER` option. Once that was added, there are 2 places in Flink that needs to respect this new case: 1. In state backends: when checking compatibility of the new serializer, if the compatibility check signals that a new reconfigured serializer instance is returned, then that said instance should be used instead of the original new serializer. 2. In composite serializers that contain nested serializers: likewise, compatibility checks on nested serializers that returns new reconfigured instances of the nested serializer should be propagated further up in the final compatibility result of the parent composite serializer. The first case is already handled in #7329. This PR deals with the second case. Since Flink has many composite serializers (e.g. `GenericArraySerializer`, `ListSerializer`, `MapSerializer`, etc.), we introduce a new base abstract class `CompositeTypeSerializerSnapshot` to encapsulate the logic of deriving the final compatibility result across multiple nested serializers. The snapshot classes of all composite serializers that were already migrated to the new `TypeSerializerSnapshot` abstractions in 1.7 are also re-visited to deduplicate code by extending the new `CompositeTypeSerializerSnapshot` class. The class-level Javadoc of `CompositeTypeSerializerSnapshot` contains more details on its serialization format and how it handles versioning for subclasses. With this PR merged, Flink will then properly support implementing state serializers with immutable configuration / state. ## Brief change log - 483dabb: Introduces the new base class `CompositeTypeSerializerSnapshot` for all composite serializers' snapshot class. - 01f58ea to 6c2ca95: Straightforward of deduplicating code in serializer snapshot classes of all composite serializers that were already migrated to the new `TypeSerializerSnapshot` abstractions in 1.7. They now extend `CompositeTypeSerializerSnapshot`. - b8310e8 to 11363c0: Replaces the original snapshot classes of `GenericArraySerializer` and `EitherSerializer` with new subclasses of `CompositeTypeSerializerSnapshot`. These two snapshot classes are handled a bit differently (i.e. introducing new classes) comparing to other snapshot classes, because the original `GenericArraySerializerConfigSnapshot` and `EitherSerializerSnapshot` implementations have more complicated versioning branches on the read path that does not allow us to change them to directly extend `CompositeTypeSerializerSnapshot`. - 55e3097: Rename the old `CompositeSerializerSnapshot` to `NestedSerializersSnapshotDelegate`, and mark it as intended for internal usage only. We now encourage users to use `CompositeTypeSerializerSnapshot` instead if they are implementing a composite serializer. ## Verifying this change New tests: `CompositeTypeSerializerSnapshotTest`. Existing relevant tests: All subclasses of `TypeSerializerSnapshotMigrationTestBase`. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (**yes** / no) - **CompositeSerializerSnapshot is renamed and marked as internal (was PublicEvolving)** - The serializers: (**yes** / no / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (**yes** / no / don't know) - The S3 file system connector: (yes / **no** / don't know)
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
