+1 to everything Max said. Gyula
On Wed, 26 Apr 2023 at 11:42, Maximilian Michels <m...@apache.org> wrote: > Thanks for starting the discussion, Jark and Xingtong! > > Flink 2.0 is long overdue. In the past, the expectations for such a > release were unreasonably high. I think everybody had a different > understanding of what exactly the criteria were. This led to releasing > 18 minor releases for the current major version. > > What I'm most excited about for Flink 2.0 is removal of baggage that > Flink has accumulated over the years: > > - Removal of Scala, deprecated interfaces, unmaintained libraries and > APIs (DataSet) > - Consolidation of configuration > - Merging of multiple scheduler implementations > - Ability to freely combine batch / streaming tasks in the runtime > > When I look at > https://docs.google.com/document/d/1_PMGl5RuDQGlV99_gL3y7OiRsF0DgCk91Coua6hFXhE/edit > , I'm a bit skeptical we will even be able to reach all these goals. I > think we have to prioritize and try to establish a deadline. Otherwise > we will end up never releasing 2.0. > > +1 on Flink 2.0 by May 2024 (not a hard deadline but I think having a > deadline helps). > > -Max > > > On Wed, Apr 26, 2023 at 10:08 AM Chesnay Schepler <ches...@apache.org> > wrote: > > > > > /Instead of defining compatibility guarantees as "this API won't > > change in all 1.x/2.x series", what if we define it as "this API won't > > change in the next 2/3 years"./ > > > > I can see some benefits to this approach (all APIs having a fixed > > minimum lifetime) but it's just gonna be difficult to communicate. > > Technically this implies that every minor release may contain breaking > > changes, which is exactly what users don't want. > > > > What problems to do you see in creating major releases every N years? > > > > > /IIUC, the milestone releases are a breakdown of the 2.0 release, > > while we are free to introduce breaking changes between them. And you > > suggest using longer-living feature branches to keep the master branch > > in a releasable state (in terms of milestone releases). Am I > > understanding it correctly?/ > > > > I think you got the general idea. There are a lot of details to be > > ironed out though (e.g., do we release connectors for each > milestone?...). > > > > Conflicts in the long-lived branches are certainly a concern, but I > > think those will be inevitable. Right now I'm not _too_ worried about > > them, at least based on my personal wish-list. > > Maybe the milestones could even help with that, as we could preemptively > > decide on an order for certain changes that have a high chance of > > conflicting with each other? > > I guess we could do that anyway. > > Maybe we should explicitly evaluate how invasive a change is (in > > relation to other breaking changes!) and manage things accordingly > > > > > > Other thoughts: > > > > We need to figure out what this release means for connectors > > compatibility-wise. The current rules for which versions a connector > > must support don't cover major releases at all. > > (This depends a bit on the scope of 2.0; if we add binary compatibility > > to Public APIs and promote a few Evolving ones then compatibility across > > minor releases becomes trivial) > > > > What process are you thinking of for deciding what breaking changes to > > make? The obvious choice would be FLIPs, but I'm worried that this will > > overload the mailing list / wiki for lots of tiny changes. > > > > Provided that we agree on doing 2.0, when would we cut the 2.0 branch? > > Would we wait a few months for people to prepare/agree on changes so we > > reduce the time we need to merge things into 2 branches? > > > > On 26/04/2023 05:51, Xintong Song wrote: > > > Thanks all for the positive feedback. > > > > > > @Martijn > > > > > > If we want to have that roadmap, should we consolidate this into a > > >> dedicated Confluence page over storing it in a Google doc? > > >> > > > Having a dedicated wiki page is definitely a good way for the roadmap > > > discussion. I haven't created one yet because it's still a proposal to > have > > > such roadmap discussion. If the community agrees with our proposal, the > > > release manager team can decide how they want to drive and track the > > > roadmap discussion. > > > > > > @Chesnay > > > > > > We should discuss how regularly we will ship major releases from now > on. > > >> Let's avoid again making breaking changes because we "gotta do it now > > >> because 3.0 isn't happening anytime soon". (e.g., every 2 years or > > >> something) > > > > > > I'm not entirely sure about shipping major releases regularly. But I do > > > agree that we may want to avoid the situation that "breaking changes > can > > > only happen now, or no idea when". Instead of defining compatibility > > > guarantees as "this API won't change in all 1.x/2.x series", what if we > > > define it as "this API won't change in the next 2/3 years". That should > > > allow us to incrementally iterate the APIs. > > > > > > E.g., in 2.a, all APIs annotated as `@Stable` will be guaranteed > compatible > > > until 2 years after 2.a is shipped, and in 2.b if the API is still > > > annotated `@Stable` it extends the compatibility guarantee to 2 years > after > > > 2.b is shipped. To remove an API, we would need to mark it as > `@Deprecated` > > > and wait for 2 years after the last release in which it was marked > > > `@Stable`. > > > > > > My thinking goes rather in the area of defining Milestone releases, > each > > >> Milestone targeting specific changes. > > >> > > > I'm trying to understand what you are suggesting here. IIUC, the > milestone > > > releases are a breakdown of the 2.0 release, while we are free to > introduce > > > breaking changes between them. And you suggest using longer-living > feature > > > branches to keep the master branch in a releasable state (in terms of > > > milestone releases). Am I understanding it correctly? > > > > > > I haven't thought this through. My gut feeling is this might be a good > > > direction to go, in terms of keeping things organized. The risk is the > cost > > > of merging feature branches and rebasing feature branches after other > > > features are merged. That depends on how close the features are > related to > > > each other. E.g., reorganization of the project modules and > dependencies > > > may change the project structure a lot, which may significantly affect > most > > > of the feature branches. Maybe we can identify such widely-affecting > > > changes and perform them at the beginning or end of the release cycle. > > > > > > Best, > > > > > > Xintong > > > > > > > > > > > > On Wed, Apr 26, 2023 at 8:23 AM ConradJam<jam.gz...@gmail.com> wrote: > > > > > >> Thanks Xintong and Jark for kicking off the great discussion! > > >> > > >> I checked the list carefully. The plans are detailed and most of the > > >> problems are covered > > >> Some of the ideas Chesnay mentioned, I think we should iterate in > > >> small steps and collect feedback in time > > >> Looking forward to the start of the work of Flink2.0, I am willing to > > >> provide assistance ~ > > >> > > >> Xintong Song<tonysong...@gmail.com> 于2023年4月25日周二 19:10写道： > > >>> Hi everyone, > > >>> > > >>> I'd like to start a discussion on planning for a Flink 2.0 release. > > >>> > > >>> AFAIK, in the past years this topic has been mentioned from time to > time, > > >>> in mailing lists, jira tickets and offline discussions. However, few > > >>> concrete steps have been taken, due to the significant determination > and > > >>> efforts it requires and distractions from other prioritized focuses. > > >> After > > >>> a series of offline discussions in the recent weeks, with folks > mostly > > >> from > > >>> our team internally as well as a few from outside Alibaba / Ververica > > >>> (thanks for insights from Becket and Robert), we believe it's time to > > >> kick > > >>> this off in the community. > > >>> > > >>> Below are some of our thoughts about the 2.0 release. Looking > forward to > > >>> your opinions and feedback. > > >>> > > >>> > > >>> ## Why plan for release 2.0? > > >>> > > >>> > > >>> Flink 1.0.0 was released in March 2016. In the past 7 years, many new > > >>> features have been added and the project has become different from > what > > >> it > > >>> used to be. So what is Flink now? What will it become in the next 3-5 > > >>> years? What do we think of Flink's position in the industry? We > believe > > >>> it's time to rethink these questions, and draw a roadmap towards > another > > >>> milestone, a milestone that worths a new major release. > > >>> > > >>> > > >>> In addition, we are still providing backwards compatibility (maybe > not > > >>> perfectly but largely) with APIs that we designed and claimed stable > 7 > > >>> years ago. While such backwards compatibility helps users to stick > with > > >> the > > >>> latest Flink releases more easily, it sometimes, and more and more > over > > >>> time, also becomes a burden for maintenance and a limitation for new > > >>> features and improvements. It's probably time to have a comprehensive > > >>> review and clean-up over all the public APIs. > > >>> > > >>> > > >>> Furthermore, next year is the 10th year for Flink as an Apache > project. > > >>> Flink joined the Apache incubator in April 2014, and became a > top-level > > >>> project in December 2014. That makes 2024 a perfect time for > bringing out > > >>> the release 2.0 milestone. And for such a major release, we'd expect > it > > >>> takes one year or even longer to prepare for, which means we probably > > >>> should start now. > > >>> > > >>> > > >>> ## What should we focus on in release 2.0? > > >>> > > >>> > > >>> - Roadmap discussion - How do we define and position Flink for > now and > > >>> in future? This is probably something we lacked. I believe some > > >> people have > > >>> thought about it, but at least it's not explicitly discussed and > > >> aligned in > > >>> the community. Ideally, the 2.0 release should be a result of the > > >> roadmap. > > >>> - Breaking changes - Important improvements, bugfixes, technical > debts > > >>> that involve breaking of API backwards compatibility, which can > only > > >> be > > >>> carried out in major releases. > > >>> - With breaking API changes, we may need multiple > 2.0-alpha/beta > > >>> versions to collect feedback. > > >>> - Key features - Significant features and improvements (e.g., > new user > > >>> stories, architectural upgrades) that may change how users use > Flink > > >> and > > >>> its position in the industry. Some items from this category may > also > > >>> involve API breaking changes or significant behavior changes. > > >>> - There are also opinions that we should stay focused as much > as > > >>> possible on the breaking changes only. Incremental / > non-breaking > > >>> improvements and features, or anything that can be added in > 2.x > > >> minor > > >>> releases, should not block the 2.0 release. > > >>> > > >>> It might be better to discuss the detailed technical items later in > > >> another > > >>> thread, to keep the current discussion focused on the overall > proposal, > > >> and > > >>> to leave time for all parties to think about their technical plans. > For > > >>> your reference, I've attached a preliminary list of work items > proposed > > >> by > > >>> Alibaba / Ververica [1]. Note that the listed items are still being > > >>> carefully evaluated and prioritized, and may change in future. > > >>> > > >>> > > >>> ## How do we manage the release? > > >>> > > >>> > > >>> #### Release Process > > >>> > > >>> > > >>> We'd expect the release process for Flink 2.0 to be different from > the > > >> 1.x > > >>> releases. > > >>> > > >>> > > >>> A major difference is that, we think the timeline-based release > > >> management > > >>> may not be suitable. The idea behind the timeline-based approach is > that > > >> we > > >>> can have more frequent releases and deliver completed features to > users > > >>> earlier, while incompleted features can be postponed to the next > release > > >>> which won't be too late with the short release cycle. However, for > > >> breaking > > >>> changes that can only take place in major releases, the price for > > >> missing a > > >>> release is too high. > > >>> > > >>> > > >>> Alternatively, we probably should discuss and agree on a list of > > >> must-have > > >>> work items. That doesn't mean keep postponing the release upon a few > > >>> delayed features. In fact, we would need to closely monitor the > progress > > >> of > > >>> the must-have items during the entire release cycle, making sure > they are > > >>> taken care of by contributors with enough expertise and capacities. > > >>> > > >>> > > >>> #### Timeline > > >>> > > >>> > > >>> The release cycle should be decided according to the feature list, > > >>> especially the must-have items that we plan to do in the release. > > >> However, > > >>> a target feature freeze date would still be helpful when making the > plan, > > >>> so that we don't pack too many things into the release. We propose > to aim > > >>> for a feature freeze around mid 2024, so that in case must-have > items are > > >>> delayed, we still have a good chance to make the release happen by > the > > >> end > > >>> of 2024. > > >>> > > >>> > > >>> #### Branch > > >>> > > >>> > > >>> A longer release cycle also means we probably should keep shiping > the 1.x > > >>> releases while preparing for the 2.0 release. We may cut a release-1 > from > > >>> master, on which we can keep developing and release 1.x releases. The > > >>> version on the master branch will then become '2.0-SNAPSHOT'. > > >>> > > >>> > > >>> #### Release Manager > > >>> > > >>> > > >>> Given the new and to-be-explored release process, longer cycle and > higher > > >>> synchronization requirements, we'd expect the 2.0 release to be more > > >>> challenging than previous 1.x releases. Therefore, we'd like to > propose > > >> to > > >>> assemble a release management team with 4-5 experienced PMC members. > Jark > > >>> and I would like to volunteer as 2 of the release managers. > > >>> > > >>> > > >>> Looking forward to your thoughts. > > >>> > > >>> > > >>> Best, > > >>> > > >>> Jark & Xintong > > >>> > > >>> > > >>> [1] > > >>> > > >> > https://docs.google.com/document/d/1_PMGl5RuDQGlV99_gL3y7OiRsF0DgCk91Coua6hFXhE/edit?usp=sharing > > >> > > >> -- > > >> Best > > >> > > >> ConradJam > > >> >
