Hi all, Here is first test for compatibility: https://github.com/apache/paimon/pull/3404
Best, Jingsong On Tue, May 28, 2024 at 12:01 PM Jingsong Li <jingsongl...@gmail.com> wrote: > > Hi Xintong, > > @Public annotation is for users. > > @StateCompatibility annotation is for the developers. Furthermore, we > can have @StorageCompatibility. > > I'm not sure if there can be some mechanism, such as automatically > collecting changed files during CI and labeling them. > > But we should add **all** compatibility tests first, and then see if > it can work very well. > > Best, > Jingsong > > On Mon, May 27, 2024 at 7:33 PM Xintong Song <tonysong...@gmail.com> wrote: > > > > +1 for providing state compatibility guarantees for Flink jobs. > > > > +1 for leveraging tests to prevent accidental state compatibility breaking > > changes, as Yong suggested. > > > > > > Jingsong, it's not very clear to me what are the semantics of the proposed > > new annotation. I'm not saying we should not introduce it. Just trying to > > be more specific. E.g., is it equivalent to `@Public` in terms of > > compatibility except that the annotated classes are not public interfaces? > > Any rule of thumb on when should developers annotate new classes with this, > > and what needs to be done when modifying a class with such an annotation? > > > > > > If this annotation is only to bring state compatibility to developers's > > attention, I wonder if the above mentioned compatibility test can already > > serve that purpose? > > > > > > Best, > > > > Xintong > > > > > > > > On Mon, May 27, 2024 at 3:55 PM Jingsong Li <jingsongl...@gmail.com> wrote: > > > > > Thanks all for your feedback. > > > > > > Yong, I think we should add tests to cover state compatibility. We > > > should store old version serialized bytes in our test resources. I > > > think unit test is OK. > > > > > > Best, > > > Jingsong > > > > > > On Mon, May 27, 2024 at 3:45 PM Yong Fang <zjur...@gmail.com> wrote: > > > > > > > > Thanks Jinsong for initiating this discussion. I agree that it's very > > > > important to upgrade the Paimon version when we restart our flink jobs > > > with > > > > the new Paimon version. Is it necessary for Paimon to introduce some > > > > unit > > > > tests or e2e tests in addition to annotations to enforce compatibility? > > > > > > > > Best, > > > > Fang Yong > > > > > > > > On Mon, May 27, 2024 at 3:25 PM yu zelin <yuzelin....@gmail.com> wrote: > > > > > > > > > Hi Jingsong, > > > > > > > > > > +1 for this. Just find a state compatibility problem: > > > > > https://github.com/apache/paimon/issues/3401. > > > > > It's important to ensure users can stop a Flink job which uses lower > > > > > version Paimon and restart it with latest version Paimon. > > > > > > > > > > Best regards, > > > > > Zelin Yu > > > > > > > > > > On Mon, May 27, 2024 at 2:32 PM Jingsong Li <jingsongl...@gmail.com> > > > > > wrote: > > > > > > > > > > > Hi Paimon Dev, > > > > > > > > > > > > We have made many changes that do not guarantee the compatibility of > > > > > > Flink job state, but in fact, we can guarantee it. We only need to > > > > > > consider the design of VersionedSerializer when making > > > > > > modifications. > > > > > > > > > > > > I plan to introduce an annotation that reminds contributors to > > > > > > ensure > > > > > > compatibility when making subsequent modifications to annotated > > > > > > classes. > > > > > > > > > > > > For example, StateCompatibility annotation. > > > > > > > > > > > > What do you think? > > > > > > > > > > > > Best, > > > > > > Jingsong > > > > > > > > > > > > > >
