Hi Jing, AFAIK most of the pain is caused by lack of backward compatibility (binary).
And to make sure I'm not adding to the confusion: It would be necessary to be able to run the iceberg connector built against Flink 1.12 with a Flink 1.13 distribution. That would solve most problems downstream projects like Iceberg, Beam etc. face today. Those projects could then publish their artifacts built against the lowest Flink version they support and rest assured users with newer versions of Flink don't run into trouble. Thanks, Thomas On Wed, Dec 29, 2021 at 8:04 AM Jing Ge <j...@ververica.com> wrote: > > Hi Piotrek, > > thanks for asking. To be honest, I hope it could be good enough if Flink > could only provide backward compatibility, which is easier than providing > forward compatibility described in the proposal. That is also one of the > reasons why I started this discussion. If, after the discussion, the > community could reach a consensus that backward compatibility is fine > enough, it is also a clear decision for us to avoid any further confusion > and complaints, i.e. make it clear that there is no guarantee for > the downstream vendors to provide backward compatibility of their own > software, they have to maintain implementations with different Flink minor > versions simultaneously. In this case, I would move the proposal about > forward compatibility to the Rejected Alternatives section. > > As far as I understood, there are at least three group of users, the first > group is the end users who write Flink job/SQL; the second group is > platform developers who use Flink to build data infrastructure and let > users in the first group submit their jobs; and the third group is > downstream software developers(vendors) who build software (module) that > depends on Flink. Platform developers in the second group will also use the > software provided by the third group in their data infra. The developers > from the second and the third groups play an important role for the > adaption/upgrade of the new Flink version in the industry. There are > compatibility issues between them, afaik, both backward and forward. I was > not able to join the discussion caused by iceberg upgrade issue previously, > after reading the whole story, building the connector with Flink 1.12 will > not solve their problem. It was the forward compatibility issue (built with > 1.13 and run on 1.12) that made it difficult for them to upgrade. > > Best regards > Jing > > On Wed, Dec 29, 2021 at 3:42 PM Piotr Nowojski <pnowoj...@apache.org> wrote: > > > Hi Jink, > > > > I haven't yet fully reviewed the FLIP document, but I wanted to clarify > > something. > > > > > Flink Forward Compatibility > > > Based on the previous clarification, Flink forward compatibility should > > mean that Flink jobs or ecosystems like external connectors/formats built > > with newer > > > Flink version Y like 1.15 can be running on an older Flink version X like > > 1.14 with no issue. With this definition, we might realize that the > > original requirement > > > in the thread [1] was asking for Flink forward compatibility, i.e. > > iceberg-flink module compiled with Flink 1.13 wanted to be running on Flink > > 1.12 cluster. And > > > from Flink user perspective, this is a backward compatibility issue. This > > is the same case: Flink forward compatibility equals Flink job/ecosystems > > backward > > > compatibility. > > > > Is it really important to provide this kind of "forward binary" > > compatibility? You are referring to the iceberg example, but the workaround > > there was quite obvious: just build the connector using Flink 1.12. As long > > as the connector is using stable, @Public API, AND assuming we provide > > binary compatibility, it should work fine with Flink 1.13. So to solve this > > user request, we could provide either forward OR backward binary > > compatibility, Right? The question is which one of those is easier to > > provide and explain to the users. > > > > Of course the big missing piece after FLIP-197 is that we haven't agreed on > > any type of binary compatibility. > > > > Best, > > Piotrek > > > > śr., 29 gru 2021 o 14:07 Jing Ge <j...@ververica.com> napisał(a): > > > > > Hi everyone, > > > > > > with great interest I have read all discussions [1][2][3] w.r.t. the > > (API?) > > > compatibility issues. The feedback coming from the Flink user's point of > > > view is very valuable. Many thanks for it. In these discussions, there > > were > > > many explanations that talked about backward and forward compatibility > > and > > > broke down to API and ABI compatibility. Since each time when a developer > > > referred to compatibility, there was an implicit context behind, which > > > might cause confusion, e.g. the forward compatibility mentioned in the > > API > > > compatibility discussion thread[1] was actually the Flink ABI backward > > > compatibility mentioned in FLIP-196. The original requirement posted in > > the > > > API compatibility discussion thread[1] was actually Flink ABI forward > > > compatibility, afaik. I will explain it in the proposal. They were all > > > correct because they talked from different perspectives. But it was hard > > > for audiences to follow it. > > > > > > I’d like to start a discussion about backward/forward compatibility from > > > both users perspective and Flink perspective. I Tried to put myself in > > > users' shoes and then to see whether we could do anything to reduce > > users' > > > effort for upgrading Flink. > > > > > > The proposal contains four parts: > > > - Clarify the definition of API and ABI, backward and forward > > compatibility > > > to make sure everyone is on the same page. > > > - Analyze and classify issues from users' feedback. > > > - Summarize the definition and the gap between the current status and > > what > > > users need. > > > - Proposed changes > > > > > > Please find the details on > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-207%3A+Flink+backward+and+forward+compatibility > > > > > > Looking forward to your feedback. Many thanks. > > > > > > best regards > > > Jing > > > > >
