Re: [DISCUSS] FLIP-263: Improve resolving schema compatibility

[Dawid Wysakowicz]

Hi Han,




However, I have two suggestions:



   TypeSerializerSnapshot {

       TypeSerializerSchemaCompatibility<T> resolveSchemaCompatibility(
            TypeSerializerSnapshot<T> oldSnapshot);

   }





What do you think?

Best,

Dawid

On 17/10/2022 04:43, Hangxiang Yu wrote:

I think in principle your proposal makes sense and the compatibility check indeed should be done in the opposite direction. 1. Should we introduce the new method to the TypeSerializerSnapshot instead? E.g. I know it has the downside that we'd need to create a snapshot for the new serializer during a restore, but the there is a lot of tooling and subclasses in the *Snapshot stack that it won't be possible to migrate to TypeSerializer stack. I have classes such as CompositeTypeSerializerSnapshot and alike in mind. 2. I'd actually be fine with breaking some external serializers and not provide a default implementation of the new method. If we don't do that we will have two methods with default implementation which call each other. It makes it also a bit harder to figure out which methods are mandatory to be implemented going forward.

Hi, Han.

Both the old method and the new method can get previous and new inner
information.

The new serializer will decide it just like the old serializer did before.

The method just specify the schema compatibility result so that other
behaviours is same as before.

On Fri, Oct 14, 2022 at 11:40 AM Han Yin <alexyin...@gmail.com> wrote:

Hi Hangxiang,

Thanks for the proposal. It seems more reasonable to let the new
serializer claim the compatibility in the cases you mentioned.

I have but one question here. What happens in the case of
“compatibleAfterMigration” after we completely reverse the direction (in
step 3)?  To be specific, migration from an old schema calls for the
previous serializer to read bytes into state objects. How should a new
serializer decide whether the migration is possible?

Best,
Han

On 2022/10/12 12:41:07 Hangxiang Yu wrote:

Dear Flink developers,

I would like to start a discussion thread on FLIP-263[1] proposing to
improve the usability of resolving schema compatibility.

Currently, the place for compatibility checks is
TypeSerializerSnapshot#resolveSchemaCompatibility
which belongs to the old serializer, There are no ways for users to

specify the

compatibility with the old serializer in the new customized serializer.

The FLIP hopes to reverse the direction of resolving schema compatibility
to improve the usability of resolving schema compatibility.

[1]

https://cwiki.apache.org/confluence/display/FLINK/FLIP-263%3A+Improve+resolving+schema+compatibility

OpenPGP_0x31D2DD10BFC15A2D.asc
Description: OpenPGP public key

OpenPGP_signature
Description: OpenPGP digital signature