Hi Xintong, @Public annotation is for users.
@StateCompatibility annotation is for the developers. Furthermore, we can have @StorageCompatibility. I'm not sure if there can be some mechanism, such as automatically collecting changed files during CI and labeling them. But we should add **all** compatibility tests first, and then see if it can work very well. Best, Jingsong On Mon, May 27, 2024 at 7:33 PM Xintong Song <tonysong...@gmail.com> wrote: > > +1 for providing state compatibility guarantees for Flink jobs. > > +1 for leveraging tests to prevent accidental state compatibility breaking > changes, as Yong suggested. > > > Jingsong, it's not very clear to me what are the semantics of the proposed > new annotation. I'm not saying we should not introduce it. Just trying to > be more specific. E.g., is it equivalent to `@Public` in terms of > compatibility except that the annotated classes are not public interfaces? > Any rule of thumb on when should developers annotate new classes with this, > and what needs to be done when modifying a class with such an annotation? > > > If this annotation is only to bring state compatibility to developers's > attention, I wonder if the above mentioned compatibility test can already > serve that purpose? > > > Best, > > Xintong > > > > On Mon, May 27, 2024 at 3:55 PM Jingsong Li <jingsongl...@gmail.com> wrote: > > > Thanks all for your feedback. > > > > Yong, I think we should add tests to cover state compatibility. We > > should store old version serialized bytes in our test resources. I > > think unit test is OK. > > > > Best, > > Jingsong > > > > On Mon, May 27, 2024 at 3:45 PM Yong Fang <zjur...@gmail.com> wrote: > > > > > > Thanks Jinsong for initiating this discussion. I agree that it's very > > > important to upgrade the Paimon version when we restart our flink jobs > > with > > > the new Paimon version. Is it necessary for Paimon to introduce some unit > > > tests or e2e tests in addition to annotations to enforce compatibility? > > > > > > Best, > > > Fang Yong > > > > > > On Mon, May 27, 2024 at 3:25 PM yu zelin <yuzelin....@gmail.com> wrote: > > > > > > > Hi Jingsong, > > > > > > > > +1 for this. Just find a state compatibility problem: > > > > https://github.com/apache/paimon/issues/3401. > > > > It's important to ensure users can stop a Flink job which uses lower > > > > version Paimon and restart it with latest version Paimon. > > > > > > > > Best regards, > > > > Zelin Yu > > > > > > > > On Mon, May 27, 2024 at 2:32 PM Jingsong Li <jingsongl...@gmail.com> > > > > wrote: > > > > > > > > > Hi Paimon Dev, > > > > > > > > > > We have made many changes that do not guarantee the compatibility of > > > > > Flink job state, but in fact, we can guarantee it. We only need to > > > > > consider the design of VersionedSerializer when making modifications. > > > > > > > > > > I plan to introduce an annotation that reminds contributors to ensure > > > > > compatibility when making subsequent modifications to annotated > > > > > classes. > > > > > > > > > > For example, StateCompatibility annotation. > > > > > > > > > > What do you think? > > > > > > > > > > Best, > > > > > Jingsong > > > > > > > > > > >
