Thanks all for your feedback. Yong, I think we should add tests to cover state compatibility. We should store old version serialized bytes in our test resources. I think unit test is OK.
Best, Jingsong On Mon, May 27, 2024 at 3:45 PM Yong Fang <zjur...@gmail.com> wrote: > > Thanks Jinsong for initiating this discussion. I agree that it's very > important to upgrade the Paimon version when we restart our flink jobs with > the new Paimon version. Is it necessary for Paimon to introduce some unit > tests or e2e tests in addition to annotations to enforce compatibility? > > Best, > Fang Yong > > On Mon, May 27, 2024 at 3:25 PM yu zelin <yuzelin....@gmail.com> wrote: > > > Hi Jingsong, > > > > +1 for this. Just find a state compatibility problem: > > https://github.com/apache/paimon/issues/3401. > > It's important to ensure users can stop a Flink job which uses lower > > version Paimon and restart it with latest version Paimon. > > > > Best regards, > > Zelin Yu > > > > On Mon, May 27, 2024 at 2:32 PM Jingsong Li <jingsongl...@gmail.com> > > wrote: > > > > > Hi Paimon Dev, > > > > > > We have made many changes that do not guarantee the compatibility of > > > Flink job state, but in fact, we can guarantee it. We only need to > > > consider the design of VersionedSerializer when making modifications. > > > > > > I plan to introduce an annotation that reminds contributors to ensure > > > compatibility when making subsequent modifications to annotated > > > classes. > > > > > > For example, StateCompatibility annotation. > > > > > > What do you think? > > > > > > Best, > > > Jingsong > > > > >
