+1 for providing state compatibility guarantees for Flink jobs. +1 for leveraging tests to prevent accidental state compatibility breaking changes, as Yong suggested.
Jingsong, it's not very clear to me what are the semantics of the proposed new annotation. I'm not saying we should not introduce it. Just trying to be more specific. E.g., is it equivalent to `@Public` in terms of compatibility except that the annotated classes are not public interfaces? Any rule of thumb on when should developers annotate new classes with this, and what needs to be done when modifying a class with such an annotation? If this annotation is only to bring state compatibility to developers's attention, I wonder if the above mentioned compatibility test can already serve that purpose? Best, Xintong On Mon, May 27, 2024 at 3:55 PM Jingsong Li <jingsongl...@gmail.com> wrote: > Thanks all for your feedback. > > Yong, I think we should add tests to cover state compatibility. We > should store old version serialized bytes in our test resources. I > think unit test is OK. > > Best, > Jingsong > > On Mon, May 27, 2024 at 3:45 PM Yong Fang <zjur...@gmail.com> wrote: > > > > Thanks Jinsong for initiating this discussion. I agree that it's very > > important to upgrade the Paimon version when we restart our flink jobs > with > > the new Paimon version. Is it necessary for Paimon to introduce some unit > > tests or e2e tests in addition to annotations to enforce compatibility? > > > > Best, > > Fang Yong > > > > On Mon, May 27, 2024 at 3:25 PM yu zelin <yuzelin....@gmail.com> wrote: > > > > > Hi Jingsong, > > > > > > +1 for this. Just find a state compatibility problem: > > > https://github.com/apache/paimon/issues/3401. > > > It's important to ensure users can stop a Flink job which uses lower > > > version Paimon and restart it with latest version Paimon. > > > > > > Best regards, > > > Zelin Yu > > > > > > On Mon, May 27, 2024 at 2:32 PM Jingsong Li <jingsongl...@gmail.com> > > > wrote: > > > > > > > Hi Paimon Dev, > > > > > > > > We have made many changes that do not guarantee the compatibility of > > > > Flink job state, but in fact, we can guarantee it. We only need to > > > > consider the design of VersionedSerializer when making modifications. > > > > > > > > I plan to introduce an annotation that reminds contributors to ensure > > > > compatibility when making subsequent modifications to annotated > > > > classes. > > > > > > > > For example, StateCompatibility annotation. > > > > > > > > What do you think? > > > > > > > > Best, > > > > Jingsong > > > > > > > >
