Thanks Becket and all for your discussion. > 1. We say this FLIP is enforced starting release 2.0. For current 1.x APIs, we provide a migration period with best effort, while allowing exceptions for immediate removal in 2.0. That means we will still try with best effort to get the ProcessFuncion API ready and deprecate the DataStream API in 1.x, but will also be allowed to remove DataStream API in 2.0 if it's not deprecated 2 minor releases before the major version bump.
> 2. We strictly follow the process in this FLIP, and will quickly bump the major version from 2.x to 3.0 once the migration period for DataStream API is reached. Sorry, I didn't read the previous detailed discussion because the discussion list was so long. I don't really like either of these options. Considering that DataStream is such an important API, can we offer a third option: 3. Maintain the DataStream API throughout 2.X and remove it until 3.x. But there's no need to assume that 2.X is a short version, it's still a normal major version. Best, Jingsong Becket Qin <becket....@gmail.com>于2023年6月22日 周四16:02写道： > Thanks much for the input, John, Stefan and Jing. > > I think Xingtong has well summarized the pros and cons of the two options. > Let's collect a few more opinions here and we can move forward with the one > more people prefer. > > Thanks, > > Jiangjie (Becket) Qin > > On Wed, Jun 21, 2023 at 3:20 AM Jing Ge <j...@ververica.com.invalid> > wrote: > > > Hi all, > > > > Thanks Xingtong for the summary. If I could only choose one of the given > > two options, I would go with option 1. I understood that option 2 worked > > great with Kafka. But the bridge release will still confuse users and my > > gut feeling is that many users will skip 2.0 and be waiting for 3.0 or > even > > 3.x. And since fewer users will use Flink 2.x, the development focus will > > be on Flink 3.0 with the fact that the current Flink release is 1.17 and > we > > are preparing 2.0 release. That is weird for me. > > > > THB, I would not name the change from @Public to @Retired as a demotion. > > The purpose of @Retire is to extend the API lifecycle with one more > stage, > > like in the real world, people born, studied, graduated, worked, and > > retired. Afaiu from the previous discussion, there are two rules we'd > like > > to follow simultaneously: > > > > 1. Public APIs can only be changed between major releases. > > 2. A smooth migration phase should be offered to users, i.e. at least 2 > > minor releases after APIs are marked as @deprecated. There should be new > > APIs as the replacement. > > > > Agree, those rules are good to improve the user friendliness. Issues we > > discussed are rising because we want to fulfill both of them. If we take > > care of deprecation very seriously, APIs can be marked as @Deprecated, > only > > when the new APIs as the replacement provide all functionalities the > > deprecated APIs have. In an ideal case without critical bugs that might > > stop users adopting the new APIs. Otherwise the expected "replacement" > will > > not happen. Users will still stick to the deprecated APIs, because the > new > > APIs can not be used. For big features, it will need at least 4 minor > > releases(ideal case), i.e. 2+ years to remove deprecated APIs: > > > > - 1st minor release to build the new APIs as the replacement and waiting > > for feedback. It might be difficult to mark the old API as deprecated in > > this release, because we are not sure if the new APIs could cover 100% > > functionalities. > > - In the lucky case, mark all old APIs as deprecated in the 2nd minor > > release. (I would even suggest having the new APIs released at least for > > two minor releases before marking it as deprecated to make sure they can > > really replace the old APIs, in case we care more about smooth migration) > > - 3rd minor release for the migration period > > - In another lucky case, the 4th release is a major release, the > > deprecated APIs could be removed. > > > > The above described scenario works only in an ideal case. In reality, it > > might take longer to get the new APIs ready and mark the old API > > deprecated. Furthermore, if the 4th release is not a major release, we > will > > have to maintain both APIs for many further minor releases. The question > is > > how to know the next major release in advance, especially 4 minor > releases' > > period, i.e. more than 2 years in advance? Given that Flink contains many > > modules, it is difficult to ask devs to create a 2-3 years deprecation > plan > > for each case. In case we want to build major releases at a fast pace, > > let's say every two years, it means devs must plan any API deprecation > > right after each major release. Afaiac, it is quite difficult. > > > > The major issue is, afaiu, if we follow rule 2, we have to keep all > @Public > > APIs, e.g. DataStream, that are not marked as deprecated yet, to 2.0. > Then > > we have to follow rule 1 to keep it unchanged until we have 3.0. That is > > why @Retired is useful to give devs more flexibility and still fulfill > both > > rules. Let's check it with examples: > > > > - we have @Public DataStream API in 1.18. It will not be marked > > as @Deprecated, because the new APIs as the replacement are not ready. > > - we keep the DataStream API itself unchanged in 2.0, but change the > > annotation from @Public to @Retire. New APIs will be introduced too. In > > this case, Rule 1 is ok, since public API is allowed to change between > > major releases. Only changing annotation is the minimal change we could > do > > and it does not break rule 1. Rule 2 is ok too, since the DataStream APIs > > work exactly the same. Attention: the change of @Public -> @Retired can > > only be done between major releases, because @Public APIs can only be > > changed between major releases. > > - in 2.1, DataStream API will be marked as deprecated. > > - in 2.2, DataStream will be kept for the migration period. > > - in 2.3, DataStream will be removed. > > > > Becket mentioned previously (please correct me if I didn't understand it > > correctly) that users might not check the changes of annotation and the > > upgrade process might not be smooth. I was wondering, if users don't pay > > attention to that, they will also not pay attention to @deprecated. The > > migration will not be smooth too even when we stick to the Java > > standard @deprecated. Let's go throw it with examples: > > > > Scenario 1: with @Retired > > > > Case 1: users upgrade from 1.18 to 2.0, 2.1, 2.2 - Nothing should be done > > for users who still want to use DataStream. The migration is smooth. But > > they should be aware of the annotation changes, i.e. @Public -> @Retired > > Case 2: users upgrade from 1.18 to 2.3 - DataStream is removed. Since it > is > > a major release upgrade, breaking changes are expected. > > Case 3: users upgrade from 2.0 to 2.3 - DataStream is removed. Since > APIs > > are marked as @Retired, breaking changes are expected too. > > > > In addition to fulfilling both rules, there are two more benefits of this > > option: > > 1. users can upgrade to 2.0 with less/no effort, i.e Case 1. > > 2. no bridge release is required. DataStream API could be deprecated in > 2.x > > without breaking any of the two rules. @Retired provides enough > flexibility > > for future deprecations and upgrades. > > > > The only issue I can find is that users might be surprised in case 3, if > > they ignore the previous @Public -> @Retired change. But for such users, > > they will ignore @deprecated annotation too and will have issues with > > pure @deprecated > > annotation too: > > > > Scenario 2: with @deprecated > > > > The example will be slightly different from scenario 1, because @Public > > APIs can only be removed between major releases. As an example, > DataStream > > API will be marked as deprecated in 1.19, kept in 1.20, and then removed > in > > 2.0 > > > > Case 1: users upgrade from 1.18 to 2.0 - APIs that are not marked as > > deprecated are removed between major releases. Rule 1 is ok. The 2 minor > > releases period defined in rule 2 sounds like a smooth migration plan, > but > > for users in this case, it is still not smooth. > > Case 2: users upgrade from 1.20 to 2.0 - Standard upgrade process with > > deprecated APIs. But for those users who ignore annotation changes, will > > have the same issue mentioned in case 1. > > > > I understood that, compared to scenario 1, users, who ignore @Public > > -> @Retired > > changes, will not face any breaking changes, after they migrated to 2.0. > > But the cost of it is bridge releases. There might be many bridge > releases > > in the future. As I mentioned previously, it is not easy to find the > right > > timing to kick off the deprecation process of APIs before the next > expected > > major release and 2-3 years in advance. > > > > Scenario 3: @PublicEvolving deprecation for users who ignore @deprecated > > annotation changes > > > > Sample API marked as deprecated in 1.19, kept in 1.20, removed in 1.21 > > > > Case 1: users upgrade from 1.18 to 1.21 - breaking change, no smooth > > migration. > > Case 2: users upgrade from 1.20 to 1.21 - same breaking change, > > since @deprecated is ignored > > > > In summary, for users who care about the annotation change, they will get > > more benefits with @Retired. For users who ignore annotation changes, > they > > will be always facing non-smooth upgrades even using pure standard > > @deprecated > > annotation/process. > > > > For the case that IDEs support @deprecated, that is true. There are no > such > > supports for homebuilt annotation. But the intention is to let users be > > aware of it. There will be other ways to do it for users who care about > > annotation changes. For users who do not care, there should be no > > difference between with tool support or without. Using @Retired with > > @deprecation > > together is at least not worse than only using @deprecation alone. And > > there are additional benefits with @Retired. > > > > Just my 2 cents and looking forward to your feedback, especially > different > > opinions that will help me understand the issue better. Thanks! > > > > > > Best regards, > > Jing > > > > > > > > > > > > > > On Tue, Jun 20, 2023 at 4:50 PM Stefan Richter > > <srich...@confluent.io.invalid> wrote: > > > > > Hi Xintong, > > > > > > Thanks for the summary, most of the points that you agreed upon also > make > > > sense to me. > > > > > > > 2. Dropping deprecated Public APIs in minor releases, or demoting > > APIs > > > > from Public to PublicEvolving / Experimental / Retired in any > version > > > bump, > > > > are not good practices. > > > > > > I hope we can can go beyond calling it “not a good practice" and reach > > > consensus that we will not demote or remove public APIs in minor > releases > > > and (technically) enforce this rule for all contributions. I think > it’s a > > > reasonable expectation for stability from a project as mature as Flink. > > > > > > > I'm personally in favor of option 1. As the migration-period rule is > > > newly > > > > proposed, I think it's fair to make exceptions for cases where we > > already > > > > missed the best chance for planning the deprecation. Moreover, I do > > > believe > > > > having a quick major version bump does confuse users. Yes, we can > > explain > > > > to users that bumping from 2.x to 3.0 does not cost anything other > than > > > the > > > > removal of an deprecated API. But having to explain this itself is an > > > > indicator that it might be confusing for users. > > > > > > > > > > > > Becket, on the other hand, prefers option 2. From my understanding, > his > > > > major point is that a quick major version bump causes barely any > actual > > > > lose on users, while in option 1 not providing the migration period > or > > > > providing a shorter on is an actual lose on users. (@Becket, please > > > correct > > > > me if I'm mistaken.) > > > > > > > > > > I would be open to both options and maybe postpone this decision until > we > > > have the complete roadmap for Flink 2.0. We would have a better > overview > > > about the technical consequences, e.g. how hard it would be to offer > and > > > maintain both APIs side by side, how well we get both APIs separated, > and > > > whether or not we will be able to use the full potential of our > breaking > > > changes before the DataStream API is completely removed. One example > for > > > the last point I can think of would be moving from lazy to eager state > > > declaration. If we believe that it is reasonable to support all planned > > > changes while maintaining both APIs, I'm leaning towards option 2. > > > > > > Best, > > > Stefan > > > > > > > > > > On 20. Jun 2023, at 14:43, Xintong Song <tonysong...@gmail.com> > wrote: > > > > > > > > Becket and I had an offline voice call earlier today. We have reached > > > > consensus on some of the arguments, but still hold different opinions > > on > > > > some others. I'd like to post the outcome here for transparency. > > > > > > > > We both agree that: > > > > > > > > 1. Providing a migration period would be beneficial for our users, > > and > > > > we should do that > > > > 2. Dropping deprecated Public APIs in minor releases, or demoting > > APIs > > > > from Public to PublicEvolving / Experimental / Retired in any > version > > > bump, > > > > are not good practices. > > > > 3. Ideally, with this FLIP, developers should be more careful and > > plan > > > > API changes ahead. That means： > > > > 1. Be more careful with designing APIs and promoting them to > > Public, > > > > so that they won't be changed / removed very soon, and also the > > > > maintenance overhead for keeping them after deprecation should > be > > > > affordable. (I believe this also aligns with John's opinion.) > > > > 2. Plan the deprecation / removal of old APIs that we don't want > > to > > > > carry to the next major version earlier, so that they'll have > the > > > > 2-minor-release migration period before the next major version > > bump. > > > > 4. A practical situation is that, we are not ready for deprecating > > the > > > > DataStream API, nor afford to carry it for another couple of years. > > > > 1. We cannot deprecate it now, because the new ProcessFunction > API > > > > (the planned replacement for DataStream API) is not yet ready. > We > > > haven't > > > > planned the ProcessFunction API earlier because the original > > > > plan was to do > > > > in-place changes directly on the DataStream API in release 2.0, > > > until the > > > > smooth migration period is proposed. Before this, the only > > > guarantees we > > > > provide for Public APIs is that they'll stay compatible until > the > > > next > > > > major version, which led to the understanding the in-place > changes > > > on > > > > Public APIs should be fine. > > > > 2. We also cannot afford carrying the DataStream API for another > > > > couple of years. One of the reasons that we want to refactor / > > > replace > > > > DataStream API is that it exposes and depends on too many > > > > Flink's internal > > > > runtime implementations, which has already limited / blocked > plans > > > for > > > > Flink's internal improvements for many times. Also due to the > wide > > > and > > > > transitive dependencies on the internals, maintaining two sets > of > > > lower > > > > APIs itself is expensive and would be better to keep it short. > > This > > > is > > > > admittedly a historical design flaw which should be avoided in > > > future. > > > > > > > > What we haven't reached consensus is about how to deal with the > current > > > > situation around the DataStream API. There are two options. > > > > > > > > > > > > 1. We say this FLIP is enforced starting release 2.0. For current > 1.x > > > > APIs, we provide a migration period with best effort, while > allowing > > > > exceptions for immediate removal in 2.0. That means we will still > try > > > with > > > > best effort to get the ProcessFuncion API ready and deprecate the > > > > DataStream API in 1.x, but will also be allowed to remove > DataStream > > > API in > > > > 2.0 if it's not deprecated 2 minor releases before the major > version > > > bump. > > > > 2. We strictly follow the process in this FLIP, and will quickly > bump > > > > the major version from 2.x to 3.0 once the migration period for > > > DataStream > > > > API is reached. > > > > > > > > I'm personally in favor of option 1. As the migration-period rule is > > > newly > > > > proposed, I think it's fair to make exceptions for cases where we > > already > > > > missed the best chance for planning the deprecation. Moreover, I do > > > believe > > > > having a quick major version bump does confuse users. Yes, we can > > explain > > > > to users that bumping from 2.x to 3.0 does not cost anything other > than > > > the > > > > removal of an deprecated API. But having to explain this itself is an > > > > indicator that it might be confusing for users. > > > > > > > > > > > > Becket, on the other hand, prefers option 2. From my understanding, > his > > > > major point is that a quick major version bump causes barely any > actual > > > > lose on users, while in option 1 not providing the migration period > or > > > > providing a shorter on is an actual lose on users. (@Becket, please > > > correct > > > > me if I'm mistaken.) > > > > > > > > > > > > And we'd like to hear more feedback from the community. > > > > > > > > > > > > Best, > > > > > > > > Xintong > > > > > > > > > > > > > > > > On Tue, Jun 20, 2023 at 2:55 AM John Roesler <vvcep...@apache.org > > > <mailto:vvcep...@apache.org>> wrote: > > > > > > > >> Hi Becket and Xintong, > > > >> > > > >> I hope you don't mind if I chime in a little. > > > >> > > > >> Once an API is marked Public, we're committing to support it until > > it's > > > >> deprecated, and once it's deprecated, to leave it in place for at > > least > > > two > > > >> minor releases and then only remove it in the following major > release. > > > >> > > > >> As a user, I would be dismayed to discover that the project > > maintainers > > > >> demoted a Public API to PublicEvolving or Experimental in order to > > > violate > > > >> the spirit of the deprecation period for Public APIs. Likewise, I'd > > > view a > > > >> rapid sequence of minor releases for the purpose of dropping > > deprecated > > > >> APIs in the next major release as another violation of the spirit of > > our > > > >> guarantees. I'm reminded of this xkcd: > > > > > > https://www.google.com/url?q=https://xkcd.com/1494/&source=gmail-imap&ust=1687869831000000&usg=AOvVaw3jFBaXa07_RvXzSLu66f6_ > > > >> > > > >> I'm glad we're gaming out these situations before we institute this > > > FLIP, > > > >> and I hope we all converge on a clear understanding of what we're > > > >> guaranteeing. From where I'm sitting, the essence of this FLIP is > > > >> specifically to prevent us from doing whatever we want and instead > to > > > >> commit to providing a suitable notice period to users who rely on > > Public > > > >> APIs. > > > >> > > > >> It honestly shouldn't matter what the maintenance overhead is (or, > > > rather, > > > >> evaluating the maintenance overhead should be a consideration before > > we > > > >> promote an API to Public to begin with). My experience in other > > > projects is > > > >> that regretting a feature so much you want to drop it asap is > > extremely > > > >> rare, and that living with the pain for a little while until the > > > >> deprecation period expires will probably make us better engineers. > > > >> > > > >> With that all said, one thing that is not a violation is to propose > a > > > new > > > >> major release in order to drop especially burdensome deprecated APIs > > > that > > > >> have already enjoyed their full deprecation period. Those really bad > > > "case > > > >> A" features will be very rare. > > > >> > > > >> Looking over the FLIP again, I see a reference to the release > > > >> documentation, but not to the build tooling support for the > Deprecated > > > >> annotation, so it might help to share this scenario: The way this > will > > > all > > > >> come together in practice is that, by giving good stability > > guarantees, > > > we > > > >> will encourage our users to bump minor versions when they are > > released, > > > >> which means that they will see compiler warnings immediately if any > of > > > the > > > >> APIs they've used are deprecated. In fact, I'd recommend for them to > > use > > > >> the `-Werror` javac flag to be sure that they don't miss those > > warnings. > > > >> Given our release cadence, this means that they will have about six > > > months > > > >> to migrate off of any deprecated APIs without being forced to avoid > > > >> updates. And they really need to be able to adopt further minor > > updates > > > in > > > >> the series, since they may contain further improvements that > actually > > > >> facilitate migrating off of the deprecated APIs. > > > >> > > > >> The compiler warnings are a far more reliable mechanism to advertise > > > >> deprecation than release notes, since users' own builds will notify > > them > > > >> right away about only the deprecations that are relevant to them, > but > > it > > > >> does rely on users feeling ok to bump up minor versions in a timely > > > >> fashion. If we consistently burn them by not observing the stability > > > >> guarantees, they'll try to avoid updates instead, and the whole > thing > > > falls > > > >> apart. > > > >> > > > >> Thanks again, > > > >> -John > > > >> > > > >> On 2023/06/19 10:43:05 Becket Qin wrote: > > > >>> Hi Xintong, > > > >>> > > > >>> Let's compare the following cases: > > > >>> > > > >>> A. If the maintenance overhead of the deprecated API is high, and > we > > > want > > > >>> to remove it after two minor releases. Then there are two options: > > > >>> A1: Demote the API in the major version bump and remove the code > > > >> with a > > > >>> minor version bump. > > > >>> A2: Do exactly the same as 1-A, except that when removing the > > code, > > > >>> bump up the major version. So this might be a short major version. > > > >>> > > > >>> B. If the maintenance overhead of the deprecated API is not high, > > > >> therefore > > > >>> keeping it for a long time is affordable. There are also two > options: > > > >>> B1: Same as A-1, demote the API in the major version bump and > > remove > > > >>> the code with a minor version bump. > > > >>> B2: Keep the API for all the minor versions in the major > version, > > > and > > > >>> only remove the code in the next major version. > > > >>> > > > >>> For case B, do we agree that B2 is the way to go? > > > >>> > > > >>> For case A: > > > >>> The following stays the same for A1 and A2: > > > >>> - users will lose the API after two minor leases. So the migration > > > >> period > > > >>> is the same for A-1 and A-2. > > > >>> - A1 and A2 will have exactly the same code after the removal of > > > >>> deprecated API, the only difference is versioning. > > > >>> - To move forward, users need to move to the next minor version in > > A1, > > > >> or > > > >>> to the next major version in A2. Because the code are the same, the > > > >> actual > > > >>> effort to upgrade is the same for A1 and A2. > > > >>> > > > >>> The differences between A-1 and A-2 are: > > > >>> - A1 allows keeping the major version release cadence. A-2 will > > have a > > > >>> short major version release. > > > >>> - A1 breaks the well understood API semantic, while A-2 does not. > > > >>> > > > >>> From what I see, since there is no well establish standard > regarding > > > how > > > >>> long a major version should be released, A short major version > > release > > > is > > > >>> potential and emotional. It is not ideal but does not have material > > > >>> downsides compared with A1. I did not hear anyone complaining about > > > Kafka > > > >>> only has two 1.x release. However, A1 actually breaks the well > > > understood > > > >>> API semantic, which has more material impact. > > > >>> > > > >>> Also, I'd imagine 90% or more of the Public APIs should fall into > > case > > > B. > > > >>> So, short major versions should be very occasional. I'd be very > > > concerned > > > >>> if the reason we choose A1 is simply because we cannot afford > > > >> maintaining a > > > >>> bunch of deprecated APIs until the next major version. This > indicates > > > >> that > > > >>> the actual problem we need to solve is to lower the maintenance > > > overhead > > > >> of > > > >>> deprecated APIs, so that we are comfortable to keep them longer. As > > > John > > > >>> and I mentioned earlier, there are ways to achieve this and we need > > to > > > >>> learn how to do it in Flink. Otherwise, our discussion about > > versioning > > > >>> here does not bring much value, because we will end up with a bunch > > of > > > >>> short-lived APIs which upset our users, no matter how we version > the > > > >>> releases. > > > >>> > > > >>> So, if there are concrete examples that you think will block us > from > > > >>> keeping API stability with affordable cost, let's take a look > > together > > > >> and > > > >>> see if that can be improved. > > > >>> > > > >>> Thanks, > > > >>> > > > >>> Jiangjie (Becket) Qin > > > >>> > > > >>> > > > >>> From what I see, > > > >>> > > > >>> > > > >>> On Mon, Jun 19, 2023 at 4:45 PM Xintong Song < > tonysong...@gmail.com > > > <mailto:tonysong...@gmail.com>> > > > >> wrote: > > > >>> > > > >>>>> > > > >>>>> The part I don't understand is if we are willing to have a > > migration > > > >>>>> period, and do a minor version bump to remove an API, what do we > > > >> lose to > > > >>>> do > > > >>>>> a major version bump instead, so we don't break the common > > versioning > > > >>>>> semantic? > > > >>>>> > > > >>>> > > > >>>> I think we are talking about the cases where a major version bump > > > >> happens > > > >>>> before a deprecated Public API reaches its migration period. So > > > >> removing > > > >>>> the API with another major version bump means we have two > > consecutive > > > >> major > > > >>>> versions in a very short time. And as previously mentioned, having > > > >> frequent > > > >>>> major version bumps would weaken the value of the commitment > "Public > > > >> API > > > >>>> stay compatible within a major version". I think users also have > > > >>>> expectations about how long a major version should live, which > > should > > > >> be at > > > >>>> least a couple of years rather than 1-2 minor releases. > > > >>>> > > > >>>> This is another option, but I think it is very likely more > expensive > > > >> than > > > >>>>> simply bumping the major version. And I imagine there will be > > > >> questions > > > >>>>> like "why this feature is in 1.20, but does not exist in 2.0"? > > > >>>>> > > > >>>> > > > >>>> Yes, it's more expensive than simply bumping the major version, > but > > > >> IMHO > > > >>>> it's probably cheaper than carrying the API until the next major > > > >> version if > > > >>>> we don't bump the major version very soon. And see my reply above > on > > > >> why > > > >>>> not bumping immediately. > > > >>>> > > > >>>> Best, > > > >>>> > > > >>>> Xintong > > > >>>> > > > >>>> > > > >>>> > > > >>>> On Mon, Jun 19, 2023 at 4:22 PM Becket Qin <becket....@gmail.com > > > <mailto:becket....@gmail.com>> > > > >> wrote: > > > >>>> > > > >>>>> Hi Xintong, > > > >>>>> > > > >>>>> Please see the replies below. > > > >>>>> > > > >>>>> I see your point, that users would feel surprised if they find > > > >> things no > > > >>>>>> longer work when upgrading to another 2.x minor release. > However, > > > >> I'd > > > >>>>> like > > > >>>>>> to point out that PublicEvolving APIs would have the similar > > > >> problem > > > >>>>>> anyway. So the question is, how do we catch users' attention and > > > >> make > > > >>>>> sure > > > >>>>>> they are aware that the Public APIs in 1.x may no longer be > Public > > > >> in > > > >>>>> 2.0. > > > >>>>>> There're various ways to do that, e.g., release notes, warnings > in > > > >>>> logs, > > > >>>>>> etc. > > > >>>>> > > > >>>>> First of all, I am not a fan of removing PublicEvolving APIs in > > minor > > > >>>>> version changes. Personally speaking, I want them to be also > > removed > > > >> in > > > >>>>> major version changes. > > > >>>>> > > > >>>>> Empirically, I rarely see projects with complex API semantic work > > > >> well. > > > >>>>> Many, if not most, users don't read docs / release notes / > warnings > > > >> in > > > >>>>> logs. I don't think that is their fault. Typically an application > > > >>>> developer > > > >>>>> will have to deal with dozens of libraries maintained by various > > > >>>>> communities / groups. It is just too difficult for them to keep > > > >> track of > > > >>>>> all the specific semantics each project puts there. If you think > > > >> about > > > >>>> it, > > > >>>>> how many of Flink developers read all the release notes of Guava, > > > >> Apache > > > >>>>> Commons, Apache Avro, ProtoBuf, etc, when you upgrade a version > of > > > >> them? > > > >>>>> One would probably try bumping the dependency version and see if > > > >> there is > > > >>>>> an exception and solve them case by case. And if it is a minor > > > >> version > > > >>>>> bump, one probably does not expect an exception at all. Another > > > >> example > > > >>>> is > > > >>>>> that in Flink we still have so many usage of deprecated methods > all > > > >> over > > > >>>>> the place, and I strongly doubt everyone knows when the source > code > > > >> of > > > >>>>> these methods are deprecated and when they should be removed. > > > >>>>> > > > >>>>> So, the most important principle of API is simple and intuitive. > > The > > > >>>>> versioning semantic is a simple and universally accepted API > > > >> stability > > > >>>>> standard. If we ourselves as Flink developers are relying on this > > > >> for our > > > >>>>> own dependencies. I don't think we can expect more from our > users. > > > >>>>> > > > >>>>> Another possible alternative: whenever there's a deprecated > Public > > > >> API > > > >>>> that > > > >>>>>> reaches a major version bump before the migration period, and we > > > >> also > > > >>>>> don't > > > >>>>>> want to carry it for all the next major release series, we may > > > >> consider > > > >>>>>> releasing more minor releases for the previous major version > after > > > >> the > > > >>>>>> bump. E.g., an Public API is deprecated in 1.19, and then we > bump > > > >> to > > > >>>> 2.0, > > > >>>>>> we can release one more 1.20 after 2.0. That should provide > users > > > >>>> another > > > >>>>>> choice rather than upgrading to 2.0, while satisfying the > > > >>>> 2-minor-release > > > >>>>>> migration period. > > > >>>>> > > > >>>>> This is another option, but I think it is very likely more > > expensive > > > >> than > > > >>>>> simply bumping the major version. And I imagine there will be > > > >> questions > > > >>>>> like "why this feature is in 1.20, but does not exist in 2.0"? > > > >>>>> > > > >>>>> I think my major point is, we should not carry APIs deprecated > in a > > > >>>>>> previous major version along all the next major version series. > > I'd > > > >>>> like > > > >>>>> to > > > >>>>>> try giving users more commitments, i.e. the migration period, as > > > >> long > > > >>>> as > > > >>>>> it > > > >>>>>> does not prevent us from making breaking changes. If it doesn't > > > >> work, > > > >>>> I'd > > > >>>>>> be in favor of not providing the migration period, but fallback > to > > > >> only > > > >>>>>> guarantee the compatibility within the major version. > > > >>>>> > > > >>>>> The part I don't understand is if we are willing to have a > > migration > > > >>>>> period, and do a minor version bump to remove an API, what do we > > > >> lose to > > > >>>> do > > > >>>>> a major version bump instead, so we don't break the common > > versioning > > > >>>>> semantic? > > > >>>>> > > > >>>>> Thanks, > > > >>>>> > > > >>>>> Jiangjie (Becket) Qin > > > >>>>> > > > >>>>> > > > >>>>> On Mon, Jun 19, 2023 at 3:20 PM Xintong Song < > > tonysong...@gmail.com > > > <mailto:tonysong...@gmail.com>> > > > >>>>> wrote: > > > >>>>> > > > >>>>>>> > > > >>>>>>> As an end user who only uses Public APIs, if I don't change my > > > >> code > > > >>>> at > > > >>>>>>> all, my expectation is the following: > > > >>>>>>> 1. Upgrading from 1.x to 2.x may have issues. > > > >>>>>>> 2. If I can upgrade from 1.x to 2.x without an issue, I am fine > > > >> with > > > >>>>> all > > > >>>>>>> the 2.x versions. > > > >>>>>>> Actually I think there are some dependency version resolution > > > >>>> policies > > > >>>>>> out > > > >>>>>>> there which picks the highest minor version when the > dependencies > > > >>>> pull > > > >>>>> in > > > >>>>>>> multiple minor versions of the same jar, which may be broken if > > > >> we > > > >>>>> remove > > > >>>>>>> the API in minor releases. > > > >>>>>>> > > > >>>>>> > > > >>>>>> I see your point, that users would feel surprised if they find > > > >> things > > > >>>> no > > > >>>>>> longer work when upgrading to another 2.x minor release. > However, > > > >> I'd > > > >>>>> like > > > >>>>>> to point out that PublicEvolving APIs would have the similar > > > >> problem > > > >>>>>> anyway. So the question is, how do we catch users' attention and > > > >> make > > > >>>>> sure > > > >>>>>> they are aware that the Public APIs in 1.x may no longer be > Public > > > >> in > > > >>>>> 2.0. > > > >>>>>> There're various ways to do that, e.g., release notes, warnings > in > > > >>>> logs, > > > >>>>>> etc. > > > >>>>>> > > > >>>>>> Another possible alternative: whenever there's a deprecated > Public > > > >> API > > > >>>>> that > > > >>>>>> reaches a major version bump before the migration period, and we > > > >> also > > > >>>>> don't > > > >>>>>> want to carry it for all the next major release series, we may > > > >> consider > > > >>>>>> releasing more minor releases for the previous major version > after > > > >> the > > > >>>>>> bump. E.g., an Public API is deprecated in 1.19, and then we > bump > > > >> to > > > >>>> 2.0, > > > >>>>>> we can release one more 1.20 after 2.0. That should provide > users > > > >>>> another > > > >>>>>> choice rather than upgrading to 2.0, while satisfying the > > > >>>> 2-minor-release > > > >>>>>> migration period. > > > >>>>>> > > > >>>>>> I think my major point is, we should not carry APIs deprecated > in > > a > > > >>>>>> previous major version along all the next major version series. > > I'd > > > >>>> like > > > >>>>> to > > > >>>>>> try giving users more commitments, i.e. the migration period, as > > > >> long > > > >>>> as > > > >>>>> it > > > >>>>>> does not prevent us from making breaking changes. If it doesn't > > > >> work, > > > >>>> I'd > > > >>>>>> be in favor of not providing the migration period, but fallback > to > > > >> only > > > >>>>>> guarantee the compatibility within the major version. > > > >>>>>> > > > >>>>>> Best, > > > >>>>>> > > > >>>>>> Xintong > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> On Mon, Jun 19, 2023 at 10:48 AM John Roesler < > > vvcep...@apache.org > > > <mailto:vvcep...@apache.org> > > > >>> > > > >>>>> wrote: > > > >>>>>> > > > >>>>>>> Hi Becket, > > > >>>>>>> > > > >>>>>>> Thanks for the reply! I’d like to continue the conversation > about > > > >>>>>>> compatibility outside of this FLIP thread, but for now, I can > > > >> accept > > > >>>>> your > > > >>>>>>> decision. It’s certainly an improvement. > > > >>>>>>> > > > >>>>>>> Thanks again, > > > >>>>>>> John > > > >>>>>>> > > > >>>>>>> On Sun, Jun 18, 2023, at 21:42, Becket Qin wrote: > > > >>>>>>>> Hi John, > > > >>>>>>>> > > > >>>>>>>> Completely agree with all you said. > > > >>>>>>>> > > > >>>>>>>> Can we consider only dropping deprecated APIs in major > releases > > > >>>>> across > > > >>>>>>> the > > > >>>>>>>>> board? I understand that Experimental and PublicEvolving APIs > > > >> are > > > >>>> by > > > >>>>>>>>> definition less stable, but it seems like this should be > > > >> reflected > > > >>>>> in > > > >>>>>>> the > > > >>>>>>>>> required deprecation period alone. I.e. that we must keep > them > > > >>>>> around > > > >>>>>>> for > > > >>>>>>>>> at least zero or one minor release, not that we can drop them > > > >> in a > > > >>>>>>> minor or > > > >>>>>>>>> patch release. > > > >>>>>>>> > > > >>>>>>>> Personally speaking, I would love to do this, for exactly the > > > >>>> reason > > > >>>>>> you > > > >>>>>>>> mentioned. However, I did not propose this due to the > following > > > >>>>>> reasons: > > > >>>>>>>> > > > >>>>>>>> 1. I am hesitating a little bit about changing the accepted > > > >> FLIPs > > > >>>> too > > > >>>>>>> soon. > > > >>>>>>>> 2. More importantly, to avoid slowing down our development. At > > > >> this > > > >>>>>>> point, > > > >>>>>>>> Flink still lacks some design / routines to support good API > > > >>>>>>> evolvability / > > > >>>>>>>> extensibility. Just like you said, it takes some time to be > > > >> good at > > > >>>>>> this. > > > >>>>>>>> In this case, my concern is that only removing Experimental / > > > >>>>>>>> PublicEvolving APIs in major version changes may result in too > > > >> much > > > >>>>>>>> overhead and dramatically slow down the development of Flink. > > > >> So, I > > > >>>>> was > > > >>>>>>>> thinking that we can start with the current status. Hopefully > > > >> after > > > >>>>> we > > > >>>>>>> are > > > >>>>>>>> more comfortable with the maintenance overhead of deprecated > > > >> APIs, > > > >>>> we > > > >>>>>> can > > > >>>>>>>> then have a stronger guarantee for Experimental / > > > >> PublicEvolving > > > >>>>> APIs. > > > >>>>>>>> > > > >>>>>>>> Thanks, > > > >>>>>>>> > > > >>>>>>>> Jiangjie (Becket) Qin > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> On Sun, Jun 18, 2023 at 6:44 AM John Roesler < > > > >> vvcep...@apache.org <mailto:vvcep...@apache.org>> > > > >>>>>>> wrote: > > > >>>>>>>> > > > >>>>>>>>> Hi Becket, > > > >>>>>>>>> > > > >>>>>>>>> Thanks for this FLIP! Having a deprecation process is really > > > >>>>>> important. > > > >>>>>>> I > > > >>>>>>>>> understand some people’s concerns about the additional burden > > > >> for > > > >>>>>>> project > > > >>>>>>>>> maintainers, but my personal experience with Kafka has been > > > >> that > > > >>>>> it’s > > > >>>>>>> very > > > >>>>>>>>> liveable and that it’s well worth the benefit to users. In > > > >> fact, > > > >>>>> users > > > >>>>>>>>> being able to confidently upgrade is also a benefit to > > > >>>> maintainers, > > > >>>>> as > > > >>>>>>> we > > > >>>>>>>>> will get fewer questions from people stuck on very old > > > >> versions. > > > >>>>>>>>> > > > >>>>>>>>> One question: > > > >>>>>>>>> Can we consider only dropping deprecated APIs in major > > > >> releases > > > >>>>> across > > > >>>>>>> the > > > >>>>>>>>> board? I understand that Experimental and PublicEvolving APIs > > > >> are > > > >>>> by > > > >>>>>>>>> definition less stable, but it seems like this should be > > > >> reflected > > > >>>>> in > > > >>>>>>> the > > > >>>>>>>>> required deprecation period alone. I.e. that we must keep > them > > > >>>>> around > > > >>>>>>> for > > > >>>>>>>>> at least zero or one minor release, not that we can drop them > > > >> in a > > > >>>>>>> minor or > > > >>>>>>>>> patch release. > > > >>>>>>>>> > > > >>>>>>>>> The advantage of forbidding the removal of any API in minor > or > > > >>>> patch > > > >>>>>>>>> releases is that users will get a strong guarantee that they > > > >> can > > > >>>>> bump > > > >>>>>>> the > > > >>>>>>>>> minor or patch version and still be able to compile, or even > > > >> just > > > >>>>>>> re-link > > > >>>>>>>>> and know that they won’t face “MethodDef” exceptions at run > > > >> time. > > > >>>>> This > > > >>>>>>> is a > > > >>>>>>>>> binary guarantee: if we allow removing even Experimental > APIs > > > >>>>> outside > > > >>>>>>> of > > > >>>>>>>>> major releases, users can no longer confidently upgrade. > > > >>>>>>>>> > > > >>>>>>>>> Aside from that, I’d share my 2 cents on a couple of points: > > > >>>>>>>>> * I’d use the official Deprecated annotation instead of > > > >>>> introducing > > > >>>>>> our > > > >>>>>>>>> own flavor (Retired, etc), since Deprecated is well > integrated > > > >>>> into > > > >>>>>>> build > > > >>>>>>>>> tools and IDEs. > > > >>>>>>>>> * I wouldn’t worry about a demotion process in this FLIP; it > > > >> seems > > > >>>>>>>>> orthogonal, and something that should probably be taken > > > >>>> case-by-case > > > >>>>>>>>> anyway. > > > >>>>>>>>> * Aside from deprecation and removal, there have been some > > > >>>>> discussions > > > >>>>>>>>> about how to evolve APIs and behavior in compatible ways. > > > >> This is > > > >>>>>>> somewhat > > > >>>>>>>>> of an art, and if folks haven’t wrestled with it before, > it’ll > > > >>>> take > > > >>>>>> some > > > >>>>>>>>> time to become good at it. I feel like this topic should also > > > >> be > > > >>>>>>> orthogonal > > > >>>>>>>>> to this FLIP, but FWIW, my suggestion would be to adopt a > > > >> simple > > > >>>>>> policy > > > >>>>>>> not > > > >>>>>>>>> to break existing user programs, and leave the “how” up to > > > >>>>>> implementers > > > >>>>>>> and > > > >>>>>>>>> reviewers. > > > >>>>>>>>> > > > >>>>>>>>> Thanks again, > > > >>>>>>>>> John > > > >>>>>>>>> > > > >>>>>>>>> On Sat, Jun 17, 2023, at 11:03, Jing Ge wrote: > > > >>>>>>>>>> Hi All, > > > >>>>>>>>>> > > > >>>>>>>>>> The @Public -> @PublicEvolving proposed by Xintong is a > > > >> great > > > >>>>> idea. > > > >>>>>>>>>> Especially, after he suggest @PublicRetired, i.e. > > > >>>> @PublicEvolving > > > >>>>>> --(2 > > > >>>>>>>>>> minor release)--> @Public --> @deprecated --(1 major > > > >>>>>>>>>> release)--> @PublicRetired. It will provide a lot of > > > >> flexibility > > > >>>>>>> without > > > >>>>>>>>>> breaking any rules we had. @Public APIs are allowed to > > > >> change > > > >>>>>> between > > > >>>>>>>>> major > > > >>>>>>>>>> releases. Changing annotations is acceptable and provides > > > >>>>> additional > > > >>>>>>>>>> tolerance i.e. user-friendliness, since the APIs themself > > > >> are > > > >>>> not > > > >>>>>>>>> changed. > > > >>>>>>>>>> > > > >>>>>>>>>> I had similar thoughts when I was facing those issues. I > > > >> want to > > > >>>>>> move > > > >>>>>>> one > > > >>>>>>>>>> step further and suggest introducing one more annotation > > > >>>> @Retired. > > > >>>>>>>>>> > > > >>>>>>>>>> Not like the @PublicRetired which is a compromise of > > > >> downgrading > > > >>>>>>> @Public > > > >>>>>>>>> to > > > >>>>>>>>>> @PublicEvolving. As I mentioned earlier in my reply, Java > > > >>>> standard > > > >>>>>>>>>> @deprecated should be used in the early stage of the > > > >> deprecation > > > >>>>>>> process > > > >>>>>>>>>> and doesn't really meet our requirement. Since Java does not > > > >>>> allow > > > >>>>>> us > > > >>>>>>> to > > > >>>>>>>>>> extend annotation, I think it would be feasible to have the > > > >> new > > > >>>>>>> @Retired > > > >>>>>>>>> to > > > >>>>>>>>>> help us monitor and manage the deprecation process, house > > > >>>>> cleaning, > > > >>>>>>> etc. > > > >>>>>>>>>> > > > >>>>>>>>>> Some ideas could be(open for discussion): > > > >>>>>>>>>> > > > >>>>>>>>>> @Retired: > > > >>>>>>>>>> > > > >>>>>>>>>> 1. There must be a replacement with functionality > > > >> compatibility > > > >>>>>> before > > > >>>>>>>>> APIs > > > >>>>>>>>>> can be marked as @Retired, i.e. DISCUSS and VOTE processes > > > >> on > > > >>>> the > > > >>>>> ML > > > >>>>>>> are > > > >>>>>>>>>> mandatory (a FLIP is recommended). > > > >>>>>>>>>> 2. APIs marked as @Retired will be removed after 1 minor > > > >> release > > > >>>>>>> sharply > > > >>>>>>>>>> (using ArchUnit to force it, needs further check whether it > > > >> is > > > >>>>>>> possible). > > > >>>>>>>>>> Devs who marked them as @Retired are responsible to remove > > > >> them. > > > >>>>>>>>>> 3. Both @Public -> @Retired and @PublicEvolving -> @Retired > > > >> are > > > >>>>>>>>>> recommended. @Experimental -> @Retired and @Internal -> > > > >> @Retired > > > >>>>>> could > > > >>>>>>>>> also > > > >>>>>>>>>> be used if it can increase user-friendliness or > > > >>>> dev-friendliness, > > > >>>>>> but > > > >>>>>>> not > > > >>>>>>>>>> mandatory. > > > >>>>>>>>>> 4. Some variables will be defined in @Retired to support the > > > >>>>>>> deprecation > > > >>>>>>>>>> process management. Further extension is possible, since the > > > >>>>>>> annotation > > > >>>>>>>>> is > > > >>>>>>>>>> built by us. > > > >>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>>>> Best regards, > > > >>>>>>>>>> Jing > > > >>>>>>>>>> > > > >>>>>>>>>> On Fri, Jun 16, 2023 at 10:31 AM Becket Qin < > > > >>>> becket....@gmail.com <mailto:becket....@gmail.com> > > > >>>>>> > > > >>>>>>>>> wrote: > > > >>>>>>>>>> > > > >>>>>>>>>>> Hi Xintong, > > > >>>>>>>>>>> > > > >>>>>>>>>>> Thanks for the explanation. Please see the replies inline > > > >>>> below. > > > >>>>>>>>>>> > > > >>>>>>>>>>> I agree. And from my understanding, demoting a Public API > > > >> is > > > >>>>> also a > > > >>>>>>>>> kind of > > > >>>>>>>>>>>> such change, just like removing one, which can only > > > >> happen > > > >>>> with > > > >>>>>>> major > > > >>>>>>>>>>>> version bumps. I'm not proposing to allow demoting Public > > > >>>> APIs > > > >>>>>>>>> anytime, > > > >>>>>>>>>>> but > > > >>>>>>>>>>>> only in the case major version bumps happen before > > > >> reaching > > > >>>> the > > > >>>>>>>>>>>> 2-minor-release migration period. Actually, demoting > > > >> would > > > >>>> be a > > > >>>>>>> weaker > > > >>>>>>>>>>>> change compared to removing the API immediately upon > > > >> major > > > >>>>>> version > > > >>>>>>>>> bumps, > > > >>>>>>>>>>>> in order to keep the commitment about the 2-minor-release > > > >>>>>> migration > > > >>>>>>>>>>> period. > > > >>>>>>>>>>>> If the concern is that `@Public` -> `@PublicEvolving` > > > >> sounds > > > >>>>>>> against > > > >>>>>>>>>>>> conventions, we may introduce a new annotation if > > > >> necessary, > > > >>>>>> e.g., > > > >>>>>>>>>>>> `@PublicRetiring`, to avoid confusions. > > > >>>>>>>>>>> > > > >>>>>>>>>>> As an end user who only uses Public APIs, if I don't > > > >> change my > > > >>>>> code > > > >>>>>>> at > > > >>>>>>>>> all, > > > >>>>>>>>>>> my expectation is the following: > > > >>>>>>>>>>> 1. Upgrading from 1.x to 2.x may have issues. > > > >>>>>>>>>>> 2. If I can upgrade from 1.x to 2.x without an issue, I am > > > >> fine > > > >>>>>> with > > > >>>>>>> all > > > >>>>>>>>>>> the 2.x versions. > > > >>>>>>>>>>> Actually I think there are some dependency version > > > >> resolution > > > >>>>>>> policies > > > >>>>>>>>> out > > > >>>>>>>>>>> there which picks the highest minor version when the > > > >>>> dependencies > > > >>>>>>> pull > > > >>>>>>>>> in > > > >>>>>>>>>>> multiple minor versions of the same jar, which may be > > > >> broken if > > > >>>>> we > > > >>>>>>>>> remove > > > >>>>>>>>>>> the API in minor releases. > > > >>>>>>>>>>> > > > >>>>>>>>>>> I'm not sure about this. Yes, it's completely "legal" that > > > >> we > > > >>>>> bump > > > >>>>>> up > > > >>>>>>>>> the > > > >>>>>>>>>>>> major version whenever a breaking change is needed. > > > >> However, > > > >>>>> this > > > >>>>>>> also > > > >>>>>>>>>>>> weakens the value of the commitment that public APIs will > > > >>>> stay > > > >>>>>>> stable > > > >>>>>>>>>>>> within the major release series, as the series can end > > > >>>> anytime. > > > >>>>>>> IMHO, > > > >>>>>>>>>>> short > > > >>>>>>>>>>>> major release series are not something "make the end > > > >> users > > > >>>>>> happy", > > > >>>>>>> but > > > >>>>>>>>>>>> backdoors that allow us as the developers to make > > > >> frequent > > > >>>>>> breaking > > > >>>>>>>>>>>> changes. On the contrary, with the demoting approach, we > > > >> can > > > >>>>>> still > > > >>>>>>>>> have > > > >>>>>>>>>>>> longer major release series, while only allowing Public > > > >> APIs > > > >>>>>>>>> deprecated > > > >>>>>>>>>>> at > > > >>>>>>>>>>>> the end of the previous major version to be removed in > > > >> the > > > >>>> next > > > >>>>>>> major > > > >>>>>>>>>>>> version. > > > >>>>>>>>>>> > > > >>>>>>>>>>> I totally agree that frequent major version bumps are not > > > >>>> ideal, > > > >>>>>> but > > > >>>>>>>>> here > > > >>>>>>>>>>> we are comparing it with a minor version bump which > > > >> removes a > > > >>>>>> Public > > > >>>>>>>>> API. > > > >>>>>>>>>>> So the context is that we have already decided to remove > > > >> this > > > >>>>>> Public > > > >>>>>>> API > > > >>>>>>>>>>> while keeping everything else backwards compatible. I > > > >> think a > > > >>>>> major > > > >>>>>>>>> version > > > >>>>>>>>>>> bump is a commonly understood signal here, compared with a > > > >>>> minor > > > >>>>>>> version > > > >>>>>>>>>>> change. From end users' perspective, for those who are not > > > >>>>>> impacted, > > > >>>>>>> in > > > >>>>>>>>>>> this case upgrading a major version is not necessarily more > > > >>>>>> involved > > > >>>>>>>>> than > > > >>>>>>>>>>> upgrading a minor version - both should be as smooth as a > > > >>>>>> dependency > > > >>>>>>>>>>> version change. For those who are impacted, they will lose > > > >> the > > > >>>>>> Public > > > >>>>>>>>> API > > > >>>>>>>>>>> anyways and a major version bump ensures there is no > > > >> surprise. > > > >>>>>>>>>>> > > > >>>>>>>>>>> Thanks, > > > >>>>>>>>>>> > > > >>>>>>>>>>> Jiangjie (Becket) Qin > > > >>>>>>>>>>> > > > >>>>>>>>>>> On Fri, Jun 16, 2023 at 10:13 AM Xintong Song < > > > >>>>>> tonysong...@gmail.com <mailto:tonysong...@gmail.com> > > > >>>>>>>> > > > >>>>>>>>>>> wrote: > > > >>>>>>>>>>> > > > >>>>>>>>>>>> Public API is a well defined common concept, and one of > > > >> its > > > >>>>>>>>>>>>> convention is that it only changes with a major version > > > >>>>> change. > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> I agree. And from my understanding, demoting a Public > > > >> API is > > > >>>>>> also a > > > >>>>>>>>> kind > > > >>>>>>>>>>>> of such change, just like removing one, which can only > > > >> happen > > > >>>>>> with > > > >>>>>>>>> major > > > >>>>>>>>>>>> version bumps. I'm not proposing to allow demoting Public > > > >>>> APIs > > > >>>>>>>>> anytime, > > > >>>>>>>>>>> but > > > >>>>>>>>>>>> only in the case major version bumps happen before > > > >> reaching > > > >>>> the > > > >>>>>>>>>>>> 2-minor-release migration period. Actually, demoting > > > >> would > > > >>>> be a > > > >>>>>>> weaker > > > >>>>>>>>>>>> change compared to removing the API immediately upon > > > >> major > > > >>>>>> version > > > >>>>>>>>> bumps, > > > >>>>>>>>>>>> in order to keep the commitment about the 2-minor-release > > > >>>>>> migration > > > >>>>>>>>>>> period. > > > >>>>>>>>>>>> If the concern is that `@Public` -> `@PublicEvolving` > > > >> sounds > > > >>>>>>> against > > > >>>>>>>>>>>> conventions, we may introduce a new annotation if > > > >> necessary, > > > >>>>>> e.g., > > > >>>>>>>>>>>> `@PublicRetiring`, to avoid confusions. > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> But it should be > > > >>>>>>>>>>>>> completely OK to bump up the major version if we really > > > >> want > > > >>>>> to > > > >>>>>>> get > > > >>>>>>>>> rid > > > >>>>>>>>>>> of > > > >>>>>>>>>>>>> a public API, right? > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> I'm not sure about this. Yes, it's completely "legal" > > > >> that we > > > >>>>>> bump > > > >>>>>>> up > > > >>>>>>>>> the > > > >>>>>>>>>>>> major version whenever a breaking change is needed. > > > >> However, > > > >>>>> this > > > >>>>>>> also > > > >>>>>>>>>>>> weakens the value of the commitment that public APIs will > > > >>>> stay > > > >>>>>>> stable > > > >>>>>>>>>>>> within the major release series, as the series can end > > > >>>> anytime. > > > >>>>>>> IMHO, > > > >>>>>>>>>>> short > > > >>>>>>>>>>>> major release series are not something "make the end > > > >> users > > > >>>>>> happy", > > > >>>>>>> but > > > >>>>>>>>>>>> backdoors that allow us as the developers to make > > > >> frequent > > > >>>>>> breaking > > > >>>>>>>>>>>> changes. On the contrary, with the demoting approach, we > > > >> can > > > >>>>>> still > > > >>>>>>>>> have > > > >>>>>>>>>>>> longer major release series, while only allowing Public > > > >> APIs > > > >>>>>>>>> deprecated > > > >>>>>>>>>>> at > > > >>>>>>>>>>>> the end of the previous major version to be removed in > > > >> the > > > >>>> next > > > >>>>>>> major > > > >>>>>>>>>>>> version. > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> Given our track record I would prefer a regular cycle > > > >> (1-2 > > > >>>>> years) > > > >>>>>>> to > > > >>>>>>>>>>>>> force us to think about this whole topic, and not put it > > > >>>> again > > > >>>>>> to > > > >>>>>>> the > > > >>>>>>>>>>>>> wayside and giving us (and users) a clear expectation on > > > >>>> when > > > >>>>>>>>> breaking > > > >>>>>>>>>>>>> changes can be made. > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> +1. I personally think 2-3 years would be a good time > > > >> for new > > > >>>>>> major > > > >>>>>>>>>>>> versions, or longer if there's no breaking changes > > > >> needed. > > > >>>> That > > > >>>>>>> makes > > > >>>>>>>>> 1-2 > > > >>>>>>>>>>>> year a perfect time to revisit the topic, while leaving > > > >> us > > > >>>> more > > > >>>>>>> time > > > >>>>>>>>> to > > > >>>>>>>>>>>> prepare the major release if needed. > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> Best, > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> Xintong > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> On Thu, Jun 15, 2023 at 10:09 PM Chesnay Schepler < > > > >>>>>>> ches...@apache.org <mailto:ches...@apache.org> > > > >>>>>>>>>> > > > >>>>>>>>>>>> wrote: > > > >>>>>>>>>>>> > > > >>>>>>>>>>>>> On 13/06/2023 17:26, Becket Qin wrote: > > > >>>>>>>>>>>>>> It would be valuable if we can avoid releasing minor > > > >>>>> versions > > > >>>>>>> for > > > >>>>>>>>>>>>> previous > > > >>>>>>>>>>>>>> major versions. > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> On paper, /absolutely /agree, but I'm not sure how > > > >> viable > > > >>>> that > > > >>>>>> is > > > >>>>>>> in > > > >>>>>>>>>>>>> practice. > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> On the current 2.0 agenda is potentially dropping > > > >> support > > > >>>> for > > > >>>>>> Java > > > >>>>>>>>> 8/11, > > > >>>>>>>>>>>>> which may very well be a problem for our current users. > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> On 13/06/2023 17:26, Becket Qin wrote: > > > >>>>>>>>>>>>>> Thanks for the feedback and sorry for the confusion > > > >> about > > > >>>>>> Public > > > >>>>>>>>> API > > > >>>>>>>>>>>>>> deprecation. I just noticed that there was a mistake > > > >> in > > > >>>> the > > > >>>>>>> NOTES > > > >>>>>>>>> part > > > >>>>>>>>>>>>> for > > > >>>>>>>>>>>>>> Public API due to a copy-paste error... I just fixed > > > >> it. > > > >>>>>>>>>>>>> I'm very relieved to hear that. Glad to hear that we > > > >> are on > > > >>>>> the > > > >>>>>>> same > > > >>>>>>>>>>>>> page on that note. > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> On 15/06/2023 15:20, Becket Qin wrote: > > > >>>>>>>>>>>>>> But it should be > > > >>>>>>>>>>>>>> completely OK to bump up the major version if we > > > >> really > > > >>>> want > > > >>>>>> to > > > >>>>>>> get > > > >>>>>>>>>>> rid > > > >>>>>>>>>>>>> of > > > >>>>>>>>>>>>>> a public API, right? > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> Technically yes, but look at how long it took to get us > > > >> to > > > >>>>> 2.0. > > > >>>>>> ;) > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> There's a separate discussion to be had on the cadence > > > >> of > > > >>>>> major > > > >>>>>>>>> releases > > > >>>>>>>>>>>>> going forward, and there seem to be different opinions > > > >> on > > > >>>>> that. > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> If we take the Kafka example of 2 minor releases between > > > >>>> major > > > >>>>>>> ones, > > > >>>>>>>>>>>>> that for us means that users have to potentially deal > > > >> with > > > >>>>>>> breaking > > > >>>>>>>>>>>>> changes every 6 months, which seems like a lot. > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> Given our track record I would prefer a regular cycle > > > >> (1-2 > > > >>>>>> years) > > > >>>>>>> to > > > >>>>>>>>>>>>> force us to think about this whole topic, and not put it > > > >>>> again > > > >>>>>> to > > > >>>>>>> the > > > >>>>>>>>>>>>> wayside and giving us (and users) a clear expectation on > > > >>>> when > > > >>>>>>>>> breaking > > > >>>>>>>>>>>>> changes can be made. > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> But again, maybe this should be in a separate thread. > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> On 14/06/2023 11:37, Becket Qin wrote: > > > >>>>>>>>>>>>>> Do you have an example of behavioral change in mind? > > > >> Not > > > >>>>> sure > > > >>>>>> I > > > >>>>>>>>> fully > > > >>>>>>>>>>>>>> understand the concern for behavioral change here. > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> This could be a lot of things. It can be performance in > > > >>>>> certain > > > >>>>>>>>>>>>> edge-cases, a bug fix that users (maybe unknowingly) > > > >> relied > > > >>>>> upon > > > >>>>>>>>>>>>> ( > > > > > > https://www.google.com/url?q=https://xkcd.com/1172/&source=gmail-imap&ust=1687869831000000&usg=AOvVaw2QkbaBd4VKyCsxxFcv1S4I > > ), > > > a semantic change to some > > > >> API. > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> For a concrete example, consider the job submission. A > > > >> few > > > >>>>>>> releases > > > >>>>>>>>> back > > > >>>>>>>>>>>>> we made changes such that the initialization of the job > > > >>>> master > > > >>>>>>>>> happens > > > >>>>>>>>>>>>> asynchronously. > > > >>>>>>>>>>>>> This meant the job submission call returns sooner, and > > > >> the > > > >>>> job > > > >>>>>>> state > > > >>>>>>>>>>>>> enum was extended to cover this state. > > > >>>>>>>>>>>>> API-wise we consider this a compatible change, but the > > > >>>>> observed > > > >>>>>>>>> behavior > > > >>>>>>>>>>>>> may be different. > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> Metrics are another example; I believe over time we > > > >> changed > > > >>>>> what > > > >>>>>>> some > > > >>>>>>>>>>>>> metrics returned a few times. > > > > > > > > >
