Hi everyone, I'm jumping in a bit late due to holidays. Generally, I agree with all the things that were shared in this thread. It would be great to see Flink 2.0 happening (especially in terms of the Java 17 support and the removal of code).
I would prefer having a major release that focuses entirely on breaking changes for the reasons Xintong mentioned in one of his posts: Feature development takes up developer/contributor's time (e.g. for implementation and review) that is not spent on issues that are relevant for that specific major release. Setting a narrower focus might help contributors prioritize their efforts in a way that benefits a faster major release. About what Chesnay and Xintong discussed in terms of API stability: > It's not necessary to introduce the breaking chagnes immediately upon > reaching the minimum guaranteed stable time. If there are multiple changes > waiting for the stable time, we can still gather them in 1 minor release. > But I see your point, from the user's perspective, the mechanism does not > provide any guarantees for the compatibility of minor releases. > I like this idea of combining the time-based deprecation with a release that collects all those breaking changes. But that seems to boil down to what we consider, by definition, a major release, doesn't it? This brings us back to the question: How many past major releases do we want to support for how long with features and/or patches? In this sense, I like what John shared in his post. I'm looking forward to the roadmap discussions. +1 on the release managers. We're happy to support the efforts if needed from Aiven's side. Thanks, Matthias On Fri, Apr 28, 2023 at 8:03 PM John Roesler <vvcep...@apache.org> wrote: > Hi all, > > Great discussion! > > For what it's worth, my experience has been that Semantic Versioning is > really the best way to think about major releases. It can occasionally be > nice to have a big "marketing release" to celebrate a major improvement, > but coupling the concept of big improvements to major version numbers makes > it hard to maintain a reasonable cadence of breaking changes. I'd be +1 on > the idea of doing more frequent major releases, and making them mostly > about compatibility breaks vs. improvements. > > At the end of the day, the nicest thing is usually to introduce new > features and improvements incrementally and compatibly, while also > deprecating unfortunate features as soon as we regret them. Once features > are deprecated, it's nice to drop them within a year or two, which implies > that you need a major release every year or two. That shouldn't frighten > us, since major releases don't guarantee non-compatibility. I.e., we > generally wouldn't break anything that hasn't already been deprecated for a > year or two, as opposed to just dropping everything that's marked as > deprecated. > > Users would only get broken if they are using features that have been > deprecated for years and many releases. Building this expectation creates a > valuable incentive to migrate off of deprecated functionality and onto the > replacements, which helps to validate the migration path while there is > still a safe fallback. It's also a significant benefit to the project's > ability to keep its API and internals clean, which ultimately benefits > users and maintainers alike. > > Independent of planning 2.0, I think it's a great idea to have a > higher-level discussion of the roadmap. Doing so will help everyone pull in > the same direction, and it'll especially help newer contributors think > about how they can contribute in valuable ways. > > Thanks! > -John > > On Fri, Apr 28, 2023, at 06:59, Jing Ge wrote: > > Hi, > > > > As far as I am concerned, it would be great to build two top level > > categories for Flink 2.0 release. > > > > 1. Future - mainly new big features or architecture improvements to > achieve > > visions like streamhouse. This makes the 2.0 release be the 2.0 instead > of > > 1.x release, i.e. the real intention of 2.0 release with a significant > > upgrade. > > 2. History - clean up tech debts, take care of histories, or whatever you > > want to name it. The main goal of this category is to take the 2.0 > release > > opportunity (since we have strong intention to do it mentioned above) to > > perform breaking changes, i.e. remove deprecated APIs and even modules, > > upgrade APIs without thinking more about backwards compatibilities, etc. > > This is kind of a buy-one-get-one benefit. In order to > "get-one"(History), > > we should, first of all, "buy-one"(Future). > > > > Best regards, > > Jing > > > > On Fri, Apr 28, 2023 at 9:57 AM Xintong Song <tonysong...@gmail.com> > wrote: > > > >> Thanks all for the positive feedback. > >> > >> So far, it seems to me that the differences of opinions are mainly > focused > >> on whether we should include non-breaking features in the 2.0 release. > >> > >> There seems to be no objections to: > >> 1. Starting to plan for the 2.0 release, with a rough timeline towards > mid > >> next year > >> 2. Becket, Jark, Martijn and Xintong as the release managers > >> 3. Initiating a project roadmap discussion > >> > >> I'll leave this discussion open for a bit longer. Also, next week is > public > >> holidays in China (I don't know if it's also in other countries). After > the > >> holidays and if there's no objections, we'll assemble the release > >> management team as discussed, and try to figure out a proper way for the > >> roadmap discussion next. > >> > >> Best, > >> > >> Xintong > >> > >> > >> > >> On Fri, Apr 28, 2023 at 3:43 PM Xintong Song <tonysong...@gmail.com> > >> wrote: > >> > >> > @Weike, > >> > > >> > Thanks for the suggestion. I think it makes sense to provide a longer > >> > supporting period for the last 1.x release. > >> > > >> > @David, > >> > > >> > I can see the benefit of focusing on breaking changes and API > clean-ups > >> > only, so the community can be more focused and possibly deliver the > 2.0 > >> > release earlier. However, I can also understand that it might be > >> > disappointing for some people (admittedly, a little bit for me as > well) > >> if > >> > the 2.0 release contains only clean-ups but no other significant > >> user-aware > >> > improvements. My personal opinion would be not to prevent people from > >> > trying to get new features into this release, but would not block the > >> > release of such features unless breaking changes are involved. > >> > > >> > @Martijn, > >> > > >> > Thanks for sharing your ideas. Glad to see that things you've listed > have > >> > a lot in common with what we put in our list. I believe that's a good > >> > signal that we share similar opinions on what is good and important > for > >> the > >> > project and the release. > >> > > >> > @Sai, > >> > > >> > Welcome to the community. And thanks for offering helps. > >> > > >> > At the moment, this discussion is only happening in this mailing > list. We > >> > may consider setting up online meetings or dedicated slack channels in > >> > future. And if so, the information will also be posted in the mailing > >> list. > >> > > >> > Best, > >> > > >> > Xintong > >> > > >> > > >> > > >> > On Fri, Apr 28, 2023 at 2:19 PM Saichandrasekar TM < > >> > saichandrase...@gmail.com> wrote: > >> > > >> >> Hi All, > >> >> > >> >> Awesome...I see this as a great opportunity for newcomers like me to > >> >> contribute. > >> >> > >> >> Is this discussion happening in a slack or discord forum too? If so, > pls > >> >> include me. > >> >> > >> >> Thanks, > >> >> Sai > >> >> > >> >> On Fri, Apr 28, 2023 at 2:55 AM Martijn Visser < > >> martijnvis...@apache.org> > >> >> wrote: > >> >> > >> >> > Hi all, > >> >> > > >> >> > I think the proposal is a good starting point. We should aim to > make > >> >> Flink > >> >> > a unified data processing, cloud friendly / cloud native > technology, > >> >> with > >> >> > proper low-level and high-level interfaces (DataStream API, Table > API, > >> >> > SQL). I think it would make a lot of sense that we write down a > vision > >> >> for > >> >> > Flink for the long term. That would also mean sharing and > discussing > >> >> more > >> >> > insights and having conversations around some of the long-term > >> direction > >> >> > from the proposal. > >> >> > > >> >> > In order to achieve that vision, I believe that we need a Flink 2.0 > >> >> which I > >> >> > consider a long overdue clean-up. That version should be the > >> foundation > >> >> for > >> >> > Flink that allows the above mentioned vision to become actual > >> proposals > >> >> and > >> >> > implementations. > >> >> > > >> >> > As a foundation in Flink 2.0, I would be inclined to say it should > be: > >> >> > > >> >> > - Remove all deprecated APIs, including the DataSet API, Scala API, > >> >> > Queryable State, legacy Source and Sink implementations, legacy SQL > >> >> > functions etc. > >> >> > - Add support for Java 17 and 21, make 17 the default (given that > the > >> >> next > >> >> > Java LTS, 21, is released in September this year and the timeline > is > >> >> set of > >> >> > 2024) > >> >> > - Drop support for Java 8 and 11 > >> >> > - Refactor the configuration layer > >> >> > - Refactor the DataStream API, such as: > >> >> > ** Having a coherent and well designed API > >> >> > ** Decouple the API into API-only modules, so no more cyclic > >> >> dependencies > >> >> > and leaking of non-APIs, including Kryo > >> >> > ** Reorganize APIs and modules > >> >> > > >> >> > I think these are some of the must-haves. Curious about the > thoughts > >> of > >> >> the > >> >> > community. > >> >> > > >> >> > Thanks, Martijn > >> >> > > >> >> > Op do 27 apr. 2023 om 10:16 schreef David Morávek <d...@apache.org > > > >> >> > > >> >> > > Hi, > >> >> > > > >> >> > > Great to see this topic moving forward; I agree it's long > overdue. > >> >> > > > >> >> > > I keep thinking about 2.0 as a chance to eliminate things that > >> didn't > >> >> > work, > >> >> > > make the feature set denser, and fix rough edges and APIs that > hold > >> us > >> >> > > back. > >> >> > > > >> >> > > Some items in the doc (Key Features section) don't tick these > boxes > >> >> for > >> >> > me, > >> >> > > as they could also be implemented in the 1x branch. We should > >> consider > >> >> > > whether we need a backward incompatible release to introduce each > >> >> > feature. > >> >> > > This should help us to keep the discussion more focused. > >> >> > > > >> >> > > Best, > >> >> > > D. > >> >> > > > >> >> > > > >> >> > > On Wed, Apr 26, 2023 at 2:33 PM DONG Weike < > kyled...@connect.hku.hk > >> > > >> >> > > wrote: > >> >> > > > >> >> > > > Hi, > >> >> > > > > >> >> > > > It is thrilling to see the foreseeable upcoming rollouts of > Flink > >> >> 2.x > >> >> > > > releases, and I believe that this roadmap can take Flink to the > >> next > >> >> > > stage > >> >> > > > of a top-of-notch unified streaming & batch computing engine. > >> >> > > > > >> >> > > > Given that all of the existing user programs are written and > run > >> in > >> >> > Flink > >> >> > > > 1.x versions as for now, and some of them are very complex and > >> rely > >> >> on > >> >> > > > various third-party connectors written with legacy APIs, one > thing > >> >> > that I > >> >> > > > have concerns about is if, one day in the future, the community > >> >> decides > >> >> > > > that new features are only given to 2.x releases, could the > last > >> >> > release > >> >> > > of > >> >> > > > Flink 1.x be converted as an LTS version (backporting severe > bug > >> >> fixes > >> >> > > and > >> >> > > > critical security patches), so that existing users could have > >> enough > >> >> > time > >> >> > > > to wait for third-party connectors to upgrade, test their > programs > >> >> on > >> >> > the > >> >> > > > Flink APIs, and avoid sudden loss of community support. > >> >> > > > > >> >> > > > Just my two cents : ) > >> >> > > > > >> >> > > > Best, > >> >> > > > Weike > >> >> > > > > >> >> > > > ________________________________ > >> >> > > > 发件人: Xintong Song <tonysong...@gmail.com> > >> >> > > > 发送时间: 2023年4月26日 20:01 > >> >> > > > 收件人: dev <dev@flink.apache.org> > >> >> > > > 主题: Re: [DISCUSS] Planning Flink 2.0 > >> >> > > > > >> >> > > > @Chesnay > >> >> > > > > >> >> > > > > >> >> > > > > Technically this implies that every minor release may contain > >> >> > breaking > >> >> > > > > changes, which is exactly what users don't want. > >> >> > > > > >> >> > > > > >> >> > > > It's not necessary to introduce the breaking chagnes > immediately > >> >> upon > >> >> > > > reaching the minimum guaranteed stable time. If there are > multiple > >> >> > > changes > >> >> > > > waiting for the stable time, we can still gather them in 1 > minor > >> >> > release. > >> >> > > > But I see your point, from the user's perspective, the > mechanism > >> >> does > >> >> > not > >> >> > > > provide any guarantees for the compatibility of minor releases. > >> >> > > > > >> >> > > > What problems to do you see in creating major releases every N > >> >> years? > >> >> > > > > > >> >> > > > > >> >> > > > It might not be concrete problem, but I'm a bit concerned by > the > >> >> > > > uncertainty. I assume N should not be too small, e.g., at > least 3. > >> >> I'd > >> >> > > > expect the decision to ship a major release would be made > based on > >> >> > > > comprehensive considerations over the situations at that time. > >> >> Making a > >> >> > > > decision now that we would ship a major release 3 years later > >> seems > >> >> a > >> >> > bit > >> >> > > > agressive to me. > >> >> > > > > >> >> > > > We need to figure out what this release means for connectors > >> >> > > > > compatibility-wise. > >> >> > > > > > >> >> > > > > >> >> > > > +1 > >> >> > > > > >> >> > > > > >> >> > > > > What process are you thinking of for deciding what breaking > >> >> changes > >> >> > to > >> >> > > > > make? The obvious choice would be FLIPs, but I'm worried that > >> this > >> >> > will > >> >> > > > > overload the mailing list / wiki for lots of tiny changes. > >> >> > > > > > >> >> > > > > >> >> > > > This should be a community decision. What I have in mind would > be: > >> >> (1) > >> >> > > > collect a wish list on wiki, (2) schedule a series of online > >> >> meetings > >> >> > > (like > >> >> > > > the release syncs) to get an agreed set of must-have items, (3) > >> >> develop > >> >> > > and > >> >> > > > polish the detailed plans of items via FLIPs, and (4) if the > plan > >> >> for a > >> >> > > > must-have item does not work out then go back to (2) for an > >> update. > >> >> I'm > >> >> > > > also open to other opinions. > >> >> > > > > >> >> > > > Would we wait a few months for people to prepare/agree on > changes > >> >> so we > >> >> > > > > reduce the time we need to merge things into 2 branches? > >> >> > > > > > >> >> > > > > >> >> > > > That's what I had in mind. Hopefully after 1.18. > >> >> > > > > >> >> > > > @Max > >> >> > > > > >> >> > > > When I look at > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> > https://docs.google.com/document/d/1_PMGl5RuDQGlV99_gL3y7OiRsF0DgCk91Coua6hFXhE/edit > >> >> > > > > , I'm a bit skeptical we will even be able to reach all these > >> >> goals. > >> >> > I > >> >> > > > > think we have to prioritize and try to establish a deadline. > >> >> > Otherwise > >> >> > > we > >> >> > > > > will end up never releasing 2.0. > >> >> > > > > >> >> > > > > >> >> > > > Sorry for the confusion. I should have explain this more > clearly. > >> We > >> >> > are > >> >> > > > not planning to finish all the items in the list. It's more > like a > >> >> > > > brainstorm, a list of candidates. We are also expecting to > collect > >> >> more > >> >> > > > ideas from the community. And after collecting the ideas, we > >> should > >> >> > > > prioritize them and decide on a subset of must-have items, > >> following > >> >> > the > >> >> > > > consensus decision making. > >> >> > > > > >> >> > > > +1 on Flink 2.0 by May 2024 (not a hard deadline but I think > >> having > >> >> a > >> >> > > > > deadline helps). > >> >> > > > > > >> >> > > > > >> >> > > > I agree that having a deadline helps. I proposed mid 2024, > which > >> is > >> >> > > similar > >> >> > > > to but not as explicit as what you proposed. We may start with > >> >> having a > >> >> > > > deadline for deciding the must-have items (e.g., by the end of > >> >> June). > >> >> > > That > >> >> > > > should make it easier for estimating the overall time needed > for > >> >> > > preparing > >> >> > > > the release. > >> >> > > > > >> >> > > > Best, > >> >> > > > > >> >> > > > Xintong > >> >> > > > > >> >> > > > > >> >> > > > > >> >> > > > On Wed, Apr 26, 2023 at 6:57 PM Gyula Fóra <gyf...@apache.org> > >> >> wrote: > >> >> > > > > >> >> > > > > +1 to everything Max said. > >> >> > > > > > >> >> > > > > Gyula > >> >> > > > > > >> >> > > > > On Wed, 26 Apr 2023 at 11:42, Maximilian Michels < > >> m...@apache.org> > >> >> > > wrote: > >> >> > > > > > >> >> > > > > > Thanks for starting the discussion, Jark and Xingtong! > >> >> > > > > > > >> >> > > > > > Flink 2.0 is long overdue. In the past, the expectations > for > >> >> such a > >> >> > > > > > release were unreasonably high. I think everybody had a > >> >> different > >> >> > > > > > understanding of what exactly the criteria were. This led > to > >> >> > > releasing > >> >> > > > > > 18 minor releases for the current major version. > >> >> > > > > > > >> >> > > > > > What I'm most excited about for Flink 2.0 is removal of > >> baggage > >> >> > that > >> >> > > > > > Flink has accumulated over the years: > >> >> > > > > > > >> >> > > > > > - Removal of Scala, deprecated interfaces, unmaintained > >> >> libraries > >> >> > and > >> >> > > > > > APIs (DataSet) > >> >> > > > > > - Consolidation of configuration > >> >> > > > > > - Merging of multiple scheduler implementations > >> >> > > > > > - Ability to freely combine batch / streaming tasks in the > >> >> runtime > >> >> > > > > > > >> >> > > > > > When I look at > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> > https://docs.google.com/document/d/1_PMGl5RuDQGlV99_gL3y7OiRsF0DgCk91Coua6hFXhE/edit > >> >> > > > > > , I'm a bit skeptical we will even be able to reach all > these > >> >> > goals. > >> >> > > I > >> >> > > > > > think we have to prioritize and try to establish a > deadline. > >> >> > > Otherwise > >> >> > > > > > we will end up never releasing 2.0. > >> >> > > > > > > >> >> > > > > > +1 on Flink 2.0 by May 2024 (not a hard deadline but I > think > >> >> > having a > >> >> > > > > > deadline helps). > >> >> > > > > > > >> >> > > > > > -Max > >> >> > > > > > > >> >> > > > > > > >> >> > > > > > On Wed, Apr 26, 2023 at 10:08 AM Chesnay Schepler < > >> >> > > ches...@apache.org> > >> >> > > > > > wrote: > >> >> > > > > > > > >> >> > > > > > > > /Instead of defining compatibility guarantees as "this > >> API > >> >> > won't > >> >> > > > > > > change in all 1.x/2.x series", what if we define it as > "this > >> >> API > >> >> > > > won't > >> >> > > > > > > change in the next 2/3 years"./ > >> >> > > > > > > > >> >> > > > > > > I can see some benefits to this approach (all APIs > having a > >> >> fixed > >> >> > > > > > > minimum lifetime) but it's just gonna be difficult to > >> >> > communicate. > >> >> > > > > > > Technically this implies that every minor release may > >> contain > >> >> > > > breaking > >> >> > > > > > > changes, which is exactly what users don't want. > >> >> > > > > > > > >> >> > > > > > > What problems to do you see in creating major releases > >> every N > >> >> > > years? > >> >> > > > > > > > >> >> > > > > > > > /IIUC, the milestone releases are a breakdown of the > 2.0 > >> >> > > release, > >> >> > > > > > > while we are free to introduce breaking changes between > >> them. > >> >> And > >> >> > > you > >> >> > > > > > > suggest using longer-living feature branches to keep the > >> >> master > >> >> > > > branch > >> >> > > > > > > in a releasable state (in terms of milestone releases). > Am I > >> >> > > > > > > understanding it correctly?/ > >> >> > > > > > > > >> >> > > > > > > I think you got the general idea. There are a lot of > details > >> >> to > >> >> > be > >> >> > > > > > > ironed out though (e.g., do we release connectors for > each > >> >> > > > > > milestone?...). > >> >> > > > > > > > >> >> > > > > > > Conflicts in the long-lived branches are certainly a > >> concern, > >> >> > but I > >> >> > > > > > > think those will be inevitable. Right now I'm not _too_ > >> >> worried > >> >> > > about > >> >> > > > > > > them, at least based on my personal wish-list. > >> >> > > > > > > Maybe the milestones could even help with that, as we > could > >> >> > > > > preemptively > >> >> > > > > > > decide on an order for certain changes that have a high > >> >> chance of > >> >> > > > > > > conflicting with each other? > >> >> > > > > > > I guess we could do that anyway. > >> >> > > > > > > Maybe we should explicitly evaluate how invasive a > change is > >> >> (in > >> >> > > > > > > relation to other breaking changes!) and manage things > >> >> > accordingly > >> >> > > > > > > > >> >> > > > > > > > >> >> > > > > > > Other thoughts: > >> >> > > > > > > > >> >> > > > > > > We need to figure out what this release means for > connectors > >> >> > > > > > > compatibility-wise. The current rules for which versions > a > >> >> > > connector > >> >> > > > > > > must support don't cover major releases at all. > >> >> > > > > > > (This depends a bit on the scope of 2.0; if we add binary > >> >> > > > compatibility > >> >> > > > > > > to Public APIs and promote a few Evolving ones then > >> >> compatibility > >> >> > > > > across > >> >> > > > > > > minor releases becomes trivial) > >> >> > > > > > > > >> >> > > > > > > What process are you thinking of for deciding what > breaking > >> >> > changes > >> >> > > > to > >> >> > > > > > > make? The obvious choice would be FLIPs, but I'm worried > >> that > >> >> > this > >> >> > > > will > >> >> > > > > > > overload the mailing list / wiki for lots of tiny > changes. > >> >> > > > > > > > >> >> > > > > > > Provided that we agree on doing 2.0, when would we cut > the > >> 2.0 > >> >> > > > branch? > >> >> > > > > > > Would we wait a few months for people to prepare/agree on > >> >> changes > >> >> > > so > >> >> > > > we > >> >> > > > > > > reduce the time we need to merge things into 2 branches? > >> >> > > > > > > > >> >> > > > > > > On 26/04/2023 05:51, Xintong Song wrote: > >> >> > > > > > > > Thanks all for the positive feedback. > >> >> > > > > > > > > >> >> > > > > > > > @Martijn > >> >> > > > > > > > > >> >> > > > > > > > If we want to have that roadmap, should we consolidate > >> this > >> >> > into > >> >> > > a > >> >> > > > > > > >> dedicated Confluence page over storing it in a Google > >> doc? > >> >> > > > > > > >> > >> >> > > > > > > > Having a dedicated wiki page is definitely a good way > for > >> >> the > >> >> > > > roadmap > >> >> > > > > > > > discussion. I haven't created one yet because it's > still a > >> >> > > proposal > >> >> > > > > to > >> >> > > > > > have > >> >> > > > > > > > such roadmap discussion. If the community agrees with > our > >> >> > > proposal, > >> >> > > > > the > >> >> > > > > > > > release manager team can decide how they want to drive > and > >> >> > track > >> >> > > > the > >> >> > > > > > > > roadmap discussion. > >> >> > > > > > > > > >> >> > > > > > > > @Chesnay > >> >> > > > > > > > > >> >> > > > > > > > We should discuss how regularly we will ship major > >> releases > >> >> > from > >> >> > > > now > >> >> > > > > > on. > >> >> > > > > > > >> Let's avoid again making breaking changes because we > >> >> "gotta do > >> >> > > it > >> >> > > > > now > >> >> > > > > > > >> because 3.0 isn't happening anytime soon". (e.g., > every 2 > >> >> > years > >> >> > > or > >> >> > > > > > > >> something) > >> >> > > > > > > > > >> >> > > > > > > > I'm not entirely sure about shipping major releases > >> >> regularly. > >> >> > > But > >> >> > > > I > >> >> > > > > do > >> >> > > > > > > > agree that we may want to avoid the situation that > >> "breaking > >> >> > > > changes > >> >> > > > > > can > >> >> > > > > > > > only happen now, or no idea when". Instead of defining > >> >> > > > compatibility > >> >> > > > > > > > guarantees as "this API won't change in all 1.x/2.x > >> series", > >> >> > what > >> >> > > > if > >> >> > > > > we > >> >> > > > > > > > define it as "this API won't change in the next 2/3 > >> years". > >> >> > That > >> >> > > > > should > >> >> > > > > > > > allow us to incrementally iterate the APIs. > >> >> > > > > > > > > >> >> > > > > > > > E.g., in 2.a, all APIs annotated as `@Stable` will be > >> >> > guaranteed > >> >> > > > > > compatible > >> >> > > > > > > > until 2 years after 2.a is shipped, and in 2.b if the > API > >> is > >> >> > > still > >> >> > > > > > > > annotated `@Stable` it extends the compatibility > guarantee > >> >> to 2 > >> >> > > > years > >> >> > > > > > after > >> >> > > > > > > > 2.b is shipped. To remove an API, we would need to > mark it > >> >> as > >> >> > > > > > `@Deprecated` > >> >> > > > > > > > and wait for 2 years after the last release in which it > >> was > >> >> > > marked > >> >> > > > > > > > `@Stable`. > >> >> > > > > > > > > >> >> > > > > > > > My thinking goes rather in the area of defining > Milestone > >> >> > > releases, > >> >> > > > > > each > >> >> > > > > > > >> Milestone targeting specific changes. > >> >> > > > > > > >> > >> >> > > > > > > > I'm trying to understand what you are suggesting here. > >> IIUC, > >> >> > the > >> >> > > > > > milestone > >> >> > > > > > > > releases are a breakdown of the 2.0 release, while we > are > >> >> free > >> >> > to > >> >> > > > > > introduce > >> >> > > > > > > > breaking changes between them. And you suggest using > >> >> > > longer-living > >> >> > > > > > feature > >> >> > > > > > > > branches to keep the master branch in a releasable > state > >> (in > >> >> > > terms > >> >> > > > of > >> >> > > > > > > > milestone releases). Am I understanding it correctly? > >> >> > > > > > > > > >> >> > > > > > > > I haven't thought this through. My gut feeling is this > >> might > >> >> > be a > >> >> > > > > good > >> >> > > > > > > > direction to go, in terms of keeping things organized. > The > >> >> risk > >> >> > > is > >> >> > > > > the > >> >> > > > > > cost > >> >> > > > > > > > of merging feature branches and rebasing feature > branches > >> >> after > >> >> > > > other > >> >> > > > > > > > features are merged. That depends on how close the > >> features > >> >> are > >> >> > > > > > related to > >> >> > > > > > > > each other. E.g., reorganization of the project modules > >> and > >> >> > > > > > dependencies > >> >> > > > > > > > may change the project structure a lot, which may > >> >> significantly > >> >> > > > > affect > >> >> > > > > > most > >> >> > > > > > > > of the feature branches. Maybe we can identify such > >> >> > > > widely-affecting > >> >> > > > > > > > changes and perform them at the beginning or end of the > >> >> release > >> >> > > > > cycle. > >> >> > > > > > > > > >> >> > > > > > > > Best, > >> >> > > > > > > > > >> >> > > > > > > > Xintong > >> >> > > > > > > > > >> >> > > > > > > > > >> >> > > > > > > > > >> >> > > > > > > > On Wed, Apr 26, 2023 at 8:23 AM ConradJam< > >> >> jam.gz...@gmail.com> > >> >> > > > > wrote: > >> >> > > > > > > > > >> >> > > > > > > >> Thanks Xintong and Jark for kicking off the great > >> >> discussion! > >> >> > > > > > > >> > >> >> > > > > > > >> I checked the list carefully. The plans are detailed > and > >> >> most > >> >> > of > >> >> > > > the > >> >> > > > > > > >> problems are covered > >> >> > > > > > > >> Some of the ideas Chesnay mentioned, I think we should > >> >> iterate > >> >> > > in > >> >> > > > > > > >> small steps and collect feedback in time > >> >> > > > > > > >> Looking forward to the start of the work of Flink2.0, > I > >> am > >> >> > > willing > >> >> > > > > to > >> >> > > > > > > >> provide assistance ~ > >> >> > > > > > > >> > >> >> > > > > > > >> Xintong Song<tonysong...@gmail.com> 于2023年4月25日周二 > >> >> 19:10写道： > >> >> > > > > > > >>> Hi everyone, > >> >> > > > > > > >>> > >> >> > > > > > > >>> I'd like to start a discussion on planning for a > Flink > >> 2.0 > >> >> > > > release. > >> >> > > > > > > >>> > >> >> > > > > > > >>> AFAIK, in the past years this topic has been > mentioned > >> >> from > >> >> > > time > >> >> > > > to > >> >> > > > > > time, > >> >> > > > > > > >>> in mailing lists, jira tickets and offline > discussions. > >> >> > > However, > >> >> > > > > few > >> >> > > > > > > >>> concrete steps have been taken, due to the > significant > >> >> > > > > determination > >> >> > > > > > and > >> >> > > > > > > >>> efforts it requires and distractions from other > >> >> prioritized > >> >> > > > > focuses. > >> >> > > > > > > >> After > >> >> > > > > > > >>> a series of offline discussions in the recent weeks, > >> with > >> >> > folks > >> >> > > > > > mostly > >> >> > > > > > > >> from > >> >> > > > > > > >>> our team internally as well as a few from outside > >> Alibaba > >> >> / > >> >> > > > > Ververica > >> >> > > > > > > >>> (thanks for insights from Becket and Robert), we > believe > >> >> it's > >> >> > > > time > >> >> > > > > to > >> >> > > > > > > >> kick > >> >> > > > > > > >>> this off in the community. > >> >> > > > > > > >>> > >> >> > > > > > > >>> Below are some of our thoughts about the 2.0 release. > >> >> Looking > >> >> > > > > > forward to > >> >> > > > > > > >>> your opinions and feedback. > >> >> > > > > > > >>> > >> >> > > > > > > >>> > >> >> > > > > > > >>> ## Why plan for release 2.0? > >> >> > > > > > > >>> > >> >> > > > > > > >>> > >> >> > > > > > > >>> Flink 1.0.0 was released in March 2016. In the past 7 > >> >> years, > >> >> > > many > >> >> > > > > new > >> >> > > > > > > >>> features have been added and the project has become > >> >> different > >> >> > > > from > >> >> > > > > > what > >> >> > > > > > > >> it > >> >> > > > > > > >>> used to be. So what is Flink now? What will it > become in > >> >> the > >> >> > > next > >> >> > > > > 3-5 > >> >> > > > > > > >>> years? What do we think of Flink's position in the > >> >> industry? > >> >> > We > >> >> > > > > > believe > >> >> > > > > > > >>> it's time to rethink these questions, and draw a > roadmap > >> >> > > towards > >> >> > > > > > another > >> >> > > > > > > >>> milestone, a milestone that worths a new major > release. > >> >> > > > > > > >>> > >> >> > > > > > > >>> > >> >> > > > > > > >>> In addition, we are still providing backwards > >> >> compatibility > >> >> > > > (maybe > >> >> > > > > > not > >> >> > > > > > > >>> perfectly but largely) with APIs that we designed and > >> >> claimed > >> >> > > > > stable > >> >> > > > > > 7 > >> >> > > > > > > >>> years ago. While such backwards compatibility helps > >> users > >> >> to > >> >> > > > stick > >> >> > > > > > with > >> >> > > > > > > >> the > >> >> > > > > > > >>> latest Flink releases more easily, it sometimes, and > >> more > >> >> and > >> >> > > > more > >> >> > > > > > over > >> >> > > > > > > >>> time, also becomes a burden for maintenance and a > >> >> limitation > >> >> > > for > >> >> > > > > new > >> >> > > > > > > >>> features and improvements. It's probably time to > have a > >> >> > > > > comprehensive > >> >> > > > > > > >>> review and clean-up over all the public APIs. > >> >> > > > > > > >>> > >> >> > > > > > > >>> > >> >> > > > > > > >>> Furthermore, next year is the 10th year for Flink as > an > >> >> > Apache > >> >> > > > > > project. > >> >> > > > > > > >>> Flink joined the Apache incubator in April 2014, and > >> >> became a > >> >> > > > > > top-level > >> >> > > > > > > >>> project in December 2014. That makes 2024 a perfect > time > >> >> for > >> >> > > > > > bringing out > >> >> > > > > > > >>> the release 2.0 milestone. And for such a major > release, > >> >> we'd > >> >> > > > > expect > >> >> > > > > > it > >> >> > > > > > > >>> takes one year or even longer to prepare for, which > >> means > >> >> we > >> >> > > > > probably > >> >> > > > > > > >>> should start now. > >> >> > > > > > > >>> > >> >> > > > > > > >>> > >> >> > > > > > > >>> ## What should we focus on in release 2.0? > >> >> > > > > > > >>> > >> >> > > > > > > >>> > >> >> > > > > > > >>> - Roadmap discussion - How do we define and > position > >> >> > Flink > >> >> > > > for > >> >> > > > > > now and > >> >> > > > > > > >>> in future? This is probably something we lacked. > I > >> >> > believe > >> >> > > > some > >> >> > > > > > > >> people have > >> >> > > > > > > >>> thought about it, but at least it's not > explicitly > >> >> > > discussed > >> >> > > > > and > >> >> > > > > > > >> aligned in > >> >> > > > > > > >>> the community. Ideally, the 2.0 release should > be a > >> >> > result > >> >> > > of > >> >> > > > > the > >> >> > > > > > > >> roadmap. > >> >> > > > > > > >>> - Breaking changes - Important improvements, > >> bugfixes, > >> >> > > > > technical > >> >> > > > > > debts > >> >> > > > > > > >>> that involve breaking of API backwards > >> compatibility, > >> >> > which > >> >> > > > can > >> >> > > > > > only > >> >> > > > > > > >> be > >> >> > > > > > > >>> carried out in major releases. > >> >> > > > > > > >>> - With breaking API changes, we may need > multiple > >> >> > > > > > 2.0-alpha/beta > >> >> > > > > > > >>> versions to collect feedback. > >> >> > > > > > > >>> - Key features - Significant features and > >> improvements > >> >> > > (e.g., > >> >> > > > > > new user > >> >> > > > > > > >>> stories, architectural upgrades) that may change > how > >> >> > users > >> >> > > > use > >> >> > > > > > Flink > >> >> > > > > > > >> and > >> >> > > > > > > >>> its position in the industry. Some items from > this > >> >> > category > >> >> > > > may > >> >> > > > > > also > >> >> > > > > > > >>> involve API breaking changes or significant > behavior > >> >> > > changes. > >> >> > > > > > > >>> - There are also opinions that we should stay > >> >> focused > >> >> > as > >> >> > > > > much > >> >> > > > > > as > >> >> > > > > > > >>> possible on the breaking changes only. > >> Incremental > >> >> / > >> >> > > > > > non-breaking > >> >> > > > > > > >>> improvements and features, or anything that > can > >> be > >> >> > added > >> >> > > > in > >> >> > > > > > 2.x > >> >> > > > > > > >> minor > >> >> > > > > > > >>> releases, should not block the 2.0 release. > >> >> > > > > > > >>> > >> >> > > > > > > >>> It might be better to discuss the detailed technical > >> items > >> >> > > later > >> >> > > > in > >> >> > > > > > > >> another > >> >> > > > > > > >>> thread, to keep the current discussion focused on the > >> >> overall > >> >> > > > > > proposal, > >> >> > > > > > > >> and > >> >> > > > > > > >>> to leave time for all parties to think about their > >> >> technical > >> >> > > > plans. > >> >> > > > > > For > >> >> > > > > > > >>> your reference, I've attached a preliminary list of > work > >> >> > items > >> >> > > > > > proposed > >> >> > > > > > > >> by > >> >> > > > > > > >>> Alibaba / Ververica [1]. Note that the listed items > are > >> >> still > >> >> > > > being > >> >> > > > > > > >>> carefully evaluated and prioritized, and may change > in > >> >> > future. > >> >> > > > > > > >>> > >> >> > > > > > > >>> > >> >> > > > > > > >>> ## How do we manage the release? > >> >> > > > > > > >>> > >> >> > > > > > > >>> > >> >> > > > > > > >>> #### Release Process > >> >> > > > > > > >>> > >> >> > > > > > > >>> > >> >> > > > > > > >>> We'd expect the release process for Flink 2.0 to be > >> >> different > >> >> > > > from > >> >> > > > > > the > >> >> > > > > > > >> 1.x > >> >> > > > > > > >>> releases. > >> >> > > > > > > >>> > >> >> > > > > > > >>> > >> >> > > > > > > >>> A major difference is that, we think the > timeline-based > >> >> > release > >> >> > > > > > > >> management > >> >> > > > > > > >>> may not be suitable. The idea behind the > timeline-based > >> >> > > approach > >> >> > > > is > >> >> > > > > > that > >> >> > > > > > > >> we > >> >> > > > > > > >>> can have more frequent releases and deliver completed > >> >> > features > >> >> > > to > >> >> > > > > > users > >> >> > > > > > > >>> earlier, while incompleted features can be postponed > to > >> >> the > >> >> > > next > >> >> > > > > > release > >> >> > > > > > > >>> which won't be too late with the short release cycle. > >> >> > However, > >> >> > > > for > >> >> > > > > > > >> breaking > >> >> > > > > > > >>> changes that can only take place in major releases, > the > >> >> price > >> >> > > for > >> >> > > > > > > >> missing a > >> >> > > > > > > >>> release is too high. > >> >> > > > > > > >>> > >> >> > > > > > > >>> > >> >> > > > > > > >>> Alternatively, we probably should discuss and agree > on a > >> >> list > >> >> > > of > >> >> > > > > > > >> must-have > >> >> > > > > > > >>> work items. That doesn't mean keep postponing the > >> release > >> >> > upon > >> >> > > a > >> >> > > > > few > >> >> > > > > > > >>> delayed features. In fact, we would need to closely > >> >> monitor > >> >> > the > >> >> > > > > > progress > >> >> > > > > > > >> of > >> >> > > > > > > >>> the must-have items during the entire release cycle, > >> >> making > >> >> > > sure > >> >> > > > > > they are > >> >> > > > > > > >>> taken care of by contributors with enough expertise > and > >> >> > > > capacities. > >> >> > > > > > > >>> > >> >> > > > > > > >>> > >> >> > > > > > > >>> #### Timeline > >> >> > > > > > > >>> > >> >> > > > > > > >>> > >> >> > > > > > > >>> The release cycle should be decided according to the > >> >> feature > >> >> > > > list, > >> >> > > > > > > >>> especially the must-have items that we plan to do in > the > >> >> > > release. > >> >> > > > > > > >> However, > >> >> > > > > > > >>> a target feature freeze date would still be helpful > when > >> >> > making > >> >> > > > the > >> >> > > > > > plan, > >> >> > > > > > > >>> so that we don't pack too many things into the > release. > >> We > >> >> > > > propose > >> >> > > > > > to aim > >> >> > > > > > > >>> for a feature freeze around mid 2024, so that in case > >> >> > must-have > >> >> > > > > > items are > >> >> > > > > > > >>> delayed, we still have a good chance to make the > release > >> >> > happen > >> >> > > > by > >> >> > > > > > the > >> >> > > > > > > >> end > >> >> > > > > > > >>> of 2024. > >> >> > > > > > > >>> > >> >> > > > > > > >>> > >> >> > > > > > > >>> #### Branch > >> >> > > > > > > >>> > >> >> > > > > > > >>> > >> >> > > > > > > >>> A longer release cycle also means we probably should > >> keep > >> >> > > shiping > >> >> > > > > > the 1.x > >> >> > > > > > > >>> releases while preparing for the 2.0 release. We may > >> cut a > >> >> > > > > release-1 > >> >> > > > > > from > >> >> > > > > > > >>> master, on which we can keep developing and release > 1.x > >> >> > > releases. > >> >> > > > > The > >> >> > > > > > > >>> version on the master branch will then become > >> >> '2.0-SNAPSHOT'. > >> >> > > > > > > >>> > >> >> > > > > > > >>> > >> >> > > > > > > >>> #### Release Manager > >> >> > > > > > > >>> > >> >> > > > > > > >>> > >> >> > > > > > > >>> Given the new and to-be-explored release process, > longer > >> >> > cycle > >> >> > > > and > >> >> > > > > > higher > >> >> > > > > > > >>> synchronization requirements, we'd expect the 2.0 > >> release > >> >> to > >> >> > be > >> >> > > > > more > >> >> > > > > > > >>> challenging than previous 1.x releases. Therefore, > we'd > >> >> like > >> >> > to > >> >> > > > > > propose > >> >> > > > > > > >> to > >> >> > > > > > > >>> assemble a release management team with 4-5 > experienced > >> >> PMC > >> >> > > > > members. > >> >> > > > > > Jark > >> >> > > > > > > >>> and I would like to volunteer as 2 of the release > >> >> managers. > >> >> > > > > > > >>> > >> >> > > > > > > >>> > >> >> > > > > > > >>> Looking forward to your thoughts. > >> >> > > > > > > >>> > >> >> > > > > > > >>> > >> >> > > > > > > >>> Best, > >> >> > > > > > > >>> > >> >> > > > > > > >>> Jark & Xintong > >> >> > > > > > > >>> > >> >> > > > > > > >>> > >> >> > > > > > > >>> [1] > >> >> > > > > > > >>> > >> >> > > > > > > >> > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> > https://docs.google.com/document/d/1_PMGl5RuDQGlV99_gL3y7OiRsF0DgCk91Coua6hFXhE/edit?usp=sharing > >> >> > > > > > > >> > >> >> > > > > > > >> -- > >> >> > > > > > > >> Best > >> >> > > > > > > >> > >> >> > > > > > > >> ConradJam > >> >> > > > > > > >> > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> > >> >> -- > >> >> Thanks, > >> >> Sai > >> >> *+91 917 623 3379* > >> >> > >> > > >> >
