I agree with Jingsong and Becket. Look at the legacy SourceFunction (a small part of DataStream API), the SourceFunction is still not and can't be marked deprecated[1] until now after the new Source was released 2 years ago, because the new Source still can't fully consume the abilities of legacy API. Considering DataStream API is the most fundamental and complex API of Flink, I think it is worth a longer time than the general process for the deprecation period to wait for the new API be mature. The above 2 options sound a bit of rush for such a widely used API.
I fully understand the concern of maintenance overhead, but it's a bit hard for others to estimate maintenance costs without a concrete design and code of the new ProcessFunction API. I agree with Becket that maybe we can re-evaluate the API deprecation process once we have the new ProcessFunction API. If the maintenance is indeed huge, I think it is reasonable to have a special rule for this case at that time. Best, Jark [1]: https://issues.apache.org/jira/browse/FLINK-28045 On Sun, 25 Jun 2023 at 16:22, Becket Qin <becket....@gmail.com> wrote: > Hi Jingsong, > > Thanks for the reply. I completely agree with you. > > The above 2 options are based on the assumption that the community cannot > afford to maintain the deprecated DataStream API for long. I'd say we > should try everything we can to maintain it for as much time as possible. > DataStream API is actually the most used API in Flink by so many users at > this point. Removing it any time soon will dramatically hurt our users. So > ideally we should keep it for at least 2 years after deprecation, if not > more. > > The prohibitively high maintenance overhead is just an assumption. > Personally speaking, I don't feel this assumption is necessarily true. We > should re-evaluate once we have the new ProcessFunction API in place. > Without the code it is hard to tell for sure. I am actually kind of > optimistic about the maintenance cost. > > Thanks, > > Jiangjie (Becket) Qin > > > > On Sun, Jun 25, 2023 at 11:30 AM Jingsong Li <jingsongl...@gmail.com> > wrote: > > > Thanks Becket and all for your discussion. > > > > > 1. We say this FLIP is enforced starting release 2.0. For current 1.x > > APIs, > > we provide a migration period with best effort, while allowing exceptions > > for immediate removal in 2.0. That means we will still try with best > effort > > to get the ProcessFuncion API ready and deprecate the DataStream API in > > 1.x, but will also be allowed to remove DataStream API in 2.0 if it's not > > deprecated 2 minor releases before the major version bump. > > > > > 2. We strictly follow the process in this FLIP, and will quickly bump > the > > major version from 2.x to 3.0 once the migration period for DataStream > API > > is reached. > > > > Sorry, I didn't read the previous detailed discussion because the > > discussion list was so long. > > > > I don't really like either of these options. > > > > Considering that DataStream is such an important API, can we offer a > third > > option: > > > > 3. Maintain the DataStream API throughout 2.X and remove it until 3.x. > But > > there's no need to assume that 2.X is a short version, it's still a > normal > > major version. > > > > Best, > > Jingsong > > > > Becket Qin <becket....@gmail.com>于2023年6月22日 周四16:02写道： > > > > > Thanks much for the input, John, Stefan and Jing. > > > > > > I think Xingtong has well summarized the pros and cons of the two > > options. > > > Let's collect a few more opinions here and we can move forward with the > > one > > > more people prefer. > > > > > > Thanks, > > > > > > Jiangjie (Becket) Qin > > > > > > On Wed, Jun 21, 2023 at 3:20 AM Jing Ge <j...@ververica.com.invalid> > > > wrote: > > > > > > > Hi all, > > > > > > > > Thanks Xingtong for the summary. If I could only choose one of the > > given > > > > two options, I would go with option 1. I understood that option 2 > > worked > > > > great with Kafka. But the bridge release will still confuse users and > > my > > > > gut feeling is that many users will skip 2.0 and be waiting for 3.0 > or > > > even > > > > 3.x. And since fewer users will use Flink 2.x, the development focus > > will > > > > be on Flink 3.0 with the fact that the current Flink release is 1.17 > > and > > > we > > > > are preparing 2.0 release. That is weird for me. > > > > > > > > THB, I would not name the change from @Public to @Retired as a > > demotion. > > > > The purpose of @Retire is to extend the API lifecycle with one more > > > stage, > > > > like in the real world, people born, studied, graduated, worked, and > > > > retired. Afaiu from the previous discussion, there are two rules we'd > > > like > > > > to follow simultaneously: > > > > > > > > 1. Public APIs can only be changed between major releases. > > > > 2. A smooth migration phase should be offered to users, i.e. at > least 2 > > > > minor releases after APIs are marked as @deprecated. There should be > > new > > > > APIs as the replacement. > > > > > > > > Agree, those rules are good to improve the user friendliness. Issues > we > > > > discussed are rising because we want to fulfill both of them. If we > > take > > > > care of deprecation very seriously, APIs can be marked as > @Deprecated, > > > only > > > > when the new APIs as the replacement provide all functionalities the > > > > deprecated APIs have. In an ideal case without critical bugs that > might > > > > stop users adopting the new APIs. Otherwise the expected > "replacement" > > > will > > > > not happen. Users will still stick to the deprecated APIs, because > the > > > new > > > > APIs can not be used. For big features, it will need at least 4 minor > > > > releases(ideal case), i.e. 2+ years to remove deprecated APIs: > > > > > > > > - 1st minor release to build the new APIs as the replacement and > > waiting > > > > for feedback. It might be difficult to mark the old API as deprecated > > in > > > > this release, because we are not sure if the new APIs could cover > 100% > > > > functionalities. > > > > - In the lucky case, mark all old APIs as deprecated in the 2nd > minor > > > > release. (I would even suggest having the new APIs released at least > > for > > > > two minor releases before marking it as deprecated to make sure they > > can > > > > really replace the old APIs, in case we care more about smooth > > migration) > > > > - 3rd minor release for the migration period > > > > - In another lucky case, the 4th release is a major release, the > > > > deprecated APIs could be removed. > > > > > > > > The above described scenario works only in an ideal case. In reality, > > it > > > > might take longer to get the new APIs ready and mark the old API > > > > deprecated. Furthermore, if the 4th release is not a major release, > we > > > will > > > > have to maintain both APIs for many further minor releases. The > > question > > > is > > > > how to know the next major release in advance, especially 4 minor > > > releases' > > > > period, i.e. more than 2 years in advance? Given that Flink contains > > many > > > > modules, it is difficult to ask devs to create a 2-3 years > deprecation > > > plan > > > > for each case. In case we want to build major releases at a fast > pace, > > > > let's say every two years, it means devs must plan any API > deprecation > > > > right after each major release. Afaiac, it is quite difficult. > > > > > > > > The major issue is, afaiu, if we follow rule 2, we have to keep all > > > @Public > > > > APIs, e.g. DataStream, that are not marked as deprecated yet, to 2.0. > > > Then > > > > we have to follow rule 1 to keep it unchanged until we have 3.0. That > > is > > > > why @Retired is useful to give devs more flexibility and still > fulfill > > > both > > > > rules. Let's check it with examples: > > > > > > > > - we have @Public DataStream API in 1.18. It will not be marked > > > > as @Deprecated, because the new APIs as the replacement are not > ready. > > > > - we keep the DataStream API itself unchanged in 2.0, but change the > > > > annotation from @Public to @Retire. New APIs will be introduced too. > In > > > > this case, Rule 1 is ok, since public API is allowed to change > between > > > > major releases. Only changing annotation is the minimal change we > could > > > do > > > > and it does not break rule 1. Rule 2 is ok too, since the DataStream > > APIs > > > > work exactly the same. Attention: the change of @Public -> @Retired > > can > > > > only be done between major releases, because @Public APIs can only be > > > > changed between major releases. > > > > - in 2.1, DataStream API will be marked as deprecated. > > > > - in 2.2, DataStream will be kept for the migration period. > > > > - in 2.3, DataStream will be removed. > > > > > > > > Becket mentioned previously (please correct me if I didn't understand > > it > > > > correctly) that users might not check the changes of annotation and > the > > > > upgrade process might not be smooth. I was wondering, if users don't > > pay > > > > attention to that, they will also not pay attention to @deprecated. > The > > > > migration will not be smooth too even when we stick to the Java > > > > standard @deprecated. Let's go throw it with examples: > > > > > > > > Scenario 1: with @Retired > > > > > > > > Case 1: users upgrade from 1.18 to 2.0, 2.1, 2.2 - Nothing should be > > done > > > > for users who still want to use DataStream. The migration is smooth. > > But > > > > they should be aware of the annotation changes, i.e. @Public -> > > @Retired > > > > Case 2: users upgrade from 1.18 to 2.3 - DataStream is removed. Since > > it > > > is > > > > a major release upgrade, breaking changes are expected. > > > > Case 3: users upgrade from 2.0 to 2.3 - DataStream is removed. Since > > > APIs > > > > are marked as @Retired, breaking changes are expected too. > > > > > > > > In addition to fulfilling both rules, there are two more benefits of > > this > > > > option: > > > > 1. users can upgrade to 2.0 with less/no effort, i.e Case 1. > > > > 2. no bridge release is required. DataStream API could be deprecated > in > > > 2.x > > > > without breaking any of the two rules. @Retired provides enough > > > flexibility > > > > for future deprecations and upgrades. > > > > > > > > The only issue I can find is that users might be surprised in case 3, > > if > > > > they ignore the previous @Public -> @Retired change. But for such > > users, > > > > they will ignore @deprecated annotation too and will have issues with > > > > pure @deprecated > > > > annotation too: > > > > > > > > Scenario 2: with @deprecated > > > > > > > > The example will be slightly different from scenario 1, because > @Public > > > > APIs can only be removed between major releases. As an example, > > > DataStream > > > > API will be marked as deprecated in 1.19, kept in 1.20, and then > > removed > > > in > > > > 2.0 > > > > > > > > Case 1: users upgrade from 1.18 to 2.0 - APIs that are not marked as > > > > deprecated are removed between major releases. Rule 1 is ok. The 2 > > minor > > > > releases period defined in rule 2 sounds like a smooth migration > plan, > > > but > > > > for users in this case, it is still not smooth. > > > > Case 2: users upgrade from 1.20 to 2.0 - Standard upgrade process > with > > > > deprecated APIs. But for those users who ignore annotation changes, > > will > > > > have the same issue mentioned in case 1. > > > > > > > > I understood that, compared to scenario 1, users, who ignore @Public > > > > -> @Retired > > > > changes, will not face any breaking changes, after they migrated to > > 2.0. > > > > But the cost of it is bridge releases. There might be many bridge > > > releases > > > > in the future. As I mentioned previously, it is not easy to find the > > > right > > > > timing to kick off the deprecation process of APIs before the next > > > expected > > > > major release and 2-3 years in advance. > > > > > > > > Scenario 3: @PublicEvolving deprecation for users who ignore > > @deprecated > > > > annotation changes > > > > > > > > Sample API marked as deprecated in 1.19, kept in 1.20, removed in > 1.21 > > > > > > > > Case 1: users upgrade from 1.18 to 1.21 - breaking change, no smooth > > > > migration. > > > > Case 2: users upgrade from 1.20 to 1.21 - same breaking change, > > > > since @deprecated is ignored > > > > > > > > In summary, for users who care about the annotation change, they will > > get > > > > more benefits with @Retired. For users who ignore annotation changes, > > > they > > > > will be always facing non-smooth upgrades even using pure standard > > > > @deprecated > > > > annotation/process. > > > > > > > > For the case that IDEs support @deprecated, that is true. There are > no > > > such > > > > supports for homebuilt annotation. But the intention is to let users > be > > > > aware of it. There will be other ways to do it for users who care > about > > > > annotation changes. For users who do not care, there should be no > > > > difference between with tool support or without. Using @Retired with > > > > @deprecation > > > > together is at least not worse than only using @deprecation alone. > And > > > > there are additional benefits with @Retired. > > > > > > > > Just my 2 cents and looking forward to your feedback, especially > > > different > > > > opinions that will help me understand the issue better. Thanks! > > > > > > > > > > > > Best regards, > > > > Jing > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Jun 20, 2023 at 4:50 PM Stefan Richter > > > > <srich...@confluent.io.invalid> wrote: > > > > > > > > > Hi Xintong, > > > > > > > > > > Thanks for the summary, most of the points that you agreed upon > also > > > make > > > > > sense to me. > > > > > > > > > > > 2. Dropping deprecated Public APIs in minor releases, or > demoting > > > > APIs > > > > > > from Public to PublicEvolving / Experimental / Retired in any > > > version > > > > > bump, > > > > > > are not good practices. > > > > > > > > > > I hope we can can go beyond calling it “not a good practice" and > > reach > > > > > consensus that we will not demote or remove public APIs in minor > > > releases > > > > > and (technically) enforce this rule for all contributions. I think > > > it’s a > > > > > reasonable expectation for stability from a project as mature as > > Flink. > > > > > > > > > > > I'm personally in favor of option 1. As the migration-period rule > > is > > > > > newly > > > > > > proposed, I think it's fair to make exceptions for cases where we > > > > already > > > > > > missed the best chance for planning the deprecation. Moreover, I > do > > > > > believe > > > > > > having a quick major version bump does confuse users. Yes, we can > > > > explain > > > > > > to users that bumping from 2.x to 3.0 does not cost anything > other > > > than > > > > > the > > > > > > removal of an deprecated API. But having to explain this itself > is > > an > > > > > > indicator that it might be confusing for users. > > > > > > > > > > > > > > > > > > Becket, on the other hand, prefers option 2. From my > understanding, > > > his > > > > > > major point is that a quick major version bump causes barely any > > > actual > > > > > > lose on users, while in option 1 not providing the migration > period > > > or > > > > > > providing a shorter on is an actual lose on users. (@Becket, > please > > > > > correct > > > > > > me if I'm mistaken.) > > > > > > > > > > > > > > > > I would be open to both options and maybe postpone this decision > > until > > > we > > > > > have the complete roadmap for Flink 2.0. We would have a better > > > overview > > > > > about the technical consequences, e.g. how hard it would be to > offer > > > and > > > > > maintain both APIs side by side, how well we get both APIs > separated, > > > and > > > > > whether or not we will be able to use the full potential of our > > > breaking > > > > > changes before the DataStream API is completely removed. One > example > > > for > > > > > the last point I can think of would be moving from lazy to eager > > state > > > > > declaration. If we believe that it is reasonable to support all > > planned > > > > > changes while maintaining both APIs, I'm leaning towards option 2. > > > > > > > > > > Best, > > > > > Stefan > > > > > > > > > > > > > > > > On 20. Jun 2023, at 14:43, Xintong Song <tonysong...@gmail.com> > > > wrote: > > > > > > > > > > > > Becket and I had an offline voice call earlier today. We have > > reached > > > > > > consensus on some of the arguments, but still hold different > > opinions > > > > on > > > > > > some others. I'd like to post the outcome here for transparency. > > > > > > > > > > > > We both agree that: > > > > > > > > > > > > 1. Providing a migration period would be beneficial for our > > users, > > > > and > > > > > > we should do that > > > > > > 2. Dropping deprecated Public APIs in minor releases, or > demoting > > > > APIs > > > > > > from Public to PublicEvolving / Experimental / Retired in any > > > version > > > > > bump, > > > > > > are not good practices. > > > > > > 3. Ideally, with this FLIP, developers should be more careful > and > > > > plan > > > > > > API changes ahead. That means： > > > > > > 1. Be more careful with designing APIs and promoting them to > > > > Public, > > > > > > so that they won't be changed / removed very soon, and also > > the > > > > > > maintenance overhead for keeping them after deprecation > should > > > be > > > > > > affordable. (I believe this also aligns with John's > opinion.) > > > > > > 2. Plan the deprecation / removal of old APIs that we don't > > want > > > > to > > > > > > carry to the next major version earlier, so that they'll > have > > > the > > > > > > 2-minor-release migration period before the next major > version > > > > bump. > > > > > > 4. A practical situation is that, we are not ready for > > deprecating > > > > the > > > > > > DataStream API, nor afford to carry it for another couple of > > years. > > > > > > 1. We cannot deprecate it now, because the new > ProcessFunction > > > API > > > > > > (the planned replacement for DataStream API) is not yet > ready. > > > We > > > > > haven't > > > > > > planned the ProcessFunction API earlier because the original > > > > > > plan was to do > > > > > > in-place changes directly on the DataStream API in release > > 2.0, > > > > > until the > > > > > > smooth migration period is proposed. Before this, the only > > > > > guarantees we > > > > > > provide for Public APIs is that they'll stay compatible > until > > > the > > > > > next > > > > > > major version, which led to the understanding the in-place > > > changes > > > > > on > > > > > > Public APIs should be fine. > > > > > > 2. We also cannot afford carrying the DataStream API for > > another > > > > > > couple of years. One of the reasons that we want to > refactor / > > > > > replace > > > > > > DataStream API is that it exposes and depends on too many > > > > > > Flink's internal > > > > > > runtime implementations, which has already limited / blocked > > > plans > > > > > for > > > > > > Flink's internal improvements for many times. Also due to > the > > > wide > > > > > and > > > > > > transitive dependencies on the internals, maintaining two > sets > > > of > > > > > lower > > > > > > APIs itself is expensive and would be better to keep it > short. > > > > This > > > > > is > > > > > > admittedly a historical design flaw which should be avoided > in > > > > > future. > > > > > > > > > > > > What we haven't reached consensus is about how to deal with the > > > current > > > > > > situation around the DataStream API. There are two options. > > > > > > > > > > > > > > > > > > 1. We say this FLIP is enforced starting release 2.0. For > current > > > 1.x > > > > > > APIs, we provide a migration period with best effort, while > > > allowing > > > > > > exceptions for immediate removal in 2.0. That means we will > still > > > try > > > > > with > > > > > > best effort to get the ProcessFuncion API ready and deprecate > the > > > > > > DataStream API in 1.x, but will also be allowed to remove > > > DataStream > > > > > API in > > > > > > 2.0 if it's not deprecated 2 minor releases before the major > > > version > > > > > bump. > > > > > > 2. We strictly follow the process in this FLIP, and will > quickly > > > bump > > > > > > the major version from 2.x to 3.0 once the migration period for > > > > > DataStream > > > > > > API is reached. > > > > > > > > > > > > I'm personally in favor of option 1. As the migration-period rule > > is > > > > > newly > > > > > > proposed, I think it's fair to make exceptions for cases where we > > > > already > > > > > > missed the best chance for planning the deprecation. Moreover, I > do > > > > > believe > > > > > > having a quick major version bump does confuse users. Yes, we can > > > > explain > > > > > > to users that bumping from 2.x to 3.0 does not cost anything > other > > > than > > > > > the > > > > > > removal of an deprecated API. But having to explain this itself > is > > an > > > > > > indicator that it might be confusing for users. > > > > > > > > > > > > > > > > > > Becket, on the other hand, prefers option 2. From my > understanding, > > > his > > > > > > major point is that a quick major version bump causes barely any > > > actual > > > > > > lose on users, while in option 1 not providing the migration > period > > > or > > > > > > providing a shorter on is an actual lose on users. (@Becket, > please > > > > > correct > > > > > > me if I'm mistaken.) > > > > > > > > > > > > > > > > > > And we'd like to hear more feedback from the community. > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > Xintong > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Jun 20, 2023 at 2:55 AM John Roesler < > vvcep...@apache.org > > > > > <mailto:vvcep...@apache.org>> wrote: > > > > > > > > > > > >> Hi Becket and Xintong, > > > > > >> > > > > > >> I hope you don't mind if I chime in a little. > > > > > >> > > > > > >> Once an API is marked Public, we're committing to support it > until > > > > it's > > > > > >> deprecated, and once it's deprecated, to leave it in place for > at > > > > least > > > > > two > > > > > >> minor releases and then only remove it in the following major > > > release. > > > > > >> > > > > > >> As a user, I would be dismayed to discover that the project > > > > maintainers > > > > > >> demoted a Public API to PublicEvolving or Experimental in order > to > > > > > violate > > > > > >> the spirit of the deprecation period for Public APIs. Likewise, > > I'd > > > > > view a > > > > > >> rapid sequence of minor releases for the purpose of dropping > > > > deprecated > > > > > >> APIs in the next major release as another violation of the > spirit > > of > > > > our > > > > > >> guarantees. I'm reminded of this xkcd: > > > > > > > > > > > > > > > https://www.google.com/url?q=https://xkcd.com/1494/&source=gmail-imap&ust=1687869831000000&usg=AOvVaw3jFBaXa07_RvXzSLu66f6_ > > > > > >> > > > > > >> I'm glad we're gaming out these situations before we institute > > this > > > > > FLIP, > > > > > >> and I hope we all converge on a clear understanding of what > we're > > > > > >> guaranteeing. From where I'm sitting, the essence of this FLIP > is > > > > > >> specifically to prevent us from doing whatever we want and > instead > > > to > > > > > >> commit to providing a suitable notice period to users who rely > on > > > > Public > > > > > >> APIs. > > > > > >> > > > > > >> It honestly shouldn't matter what the maintenance overhead is > (or, > > > > > rather, > > > > > >> evaluating the maintenance overhead should be a consideration > > before > > > > we > > > > > >> promote an API to Public to begin with). My experience in other > > > > > projects is > > > > > >> that regretting a feature so much you want to drop it asap is > > > > extremely > > > > > >> rare, and that living with the pain for a little while until the > > > > > >> deprecation period expires will probably make us better > engineers. > > > > > >> > > > > > >> With that all said, one thing that is not a violation is to > > propose > > > a > > > > > new > > > > > >> major release in order to drop especially burdensome deprecated > > APIs > > > > > that > > > > > >> have already enjoyed their full deprecation period. Those really > > bad > > > > > "case > > > > > >> A" features will be very rare. > > > > > >> > > > > > >> Looking over the FLIP again, I see a reference to the release > > > > > >> documentation, but not to the build tooling support for the > > > Deprecated > > > > > >> annotation, so it might help to share this scenario: The way > this > > > will > > > > > all > > > > > >> come together in practice is that, by giving good stability > > > > guarantees, > > > > > we > > > > > >> will encourage our users to bump minor versions when they are > > > > released, > > > > > >> which means that they will see compiler warnings immediately if > > any > > > of > > > > > the > > > > > >> APIs they've used are deprecated. In fact, I'd recommend for > them > > to > > > > use > > > > > >> the `-Werror` javac flag to be sure that they don't miss those > > > > warnings. > > > > > >> Given our release cadence, this means that they will have about > > six > > > > > months > > > > > >> to migrate off of any deprecated APIs without being forced to > > avoid > > > > > >> updates. And they really need to be able to adopt further minor > > > > updates > > > > > in > > > > > >> the series, since they may contain further improvements that > > > actually > > > > > >> facilitate migrating off of the deprecated APIs. > > > > > >> > > > > > >> The compiler warnings are a far more reliable mechanism to > > advertise > > > > > >> deprecation than release notes, since users' own builds will > > notify > > > > them > > > > > >> right away about only the deprecations that are relevant to > them, > > > but > > > > it > > > > > >> does rely on users feeling ok to bump up minor versions in a > > timely > > > > > >> fashion. If we consistently burn them by not observing the > > stability > > > > > >> guarantees, they'll try to avoid updates instead, and the whole > > > thing > > > > > falls > > > > > >> apart. > > > > > >> > > > > > >> Thanks again, > > > > > >> -John > > > > > >> > > > > > >> On 2023/06/19 10:43:05 Becket Qin wrote: > > > > > >>> Hi Xintong, > > > > > >>> > > > > > >>> Let's compare the following cases: > > > > > >>> > > > > > >>> A. If the maintenance overhead of the deprecated API is high, > and > > > we > > > > > want > > > > > >>> to remove it after two minor releases. Then there are two > > options: > > > > > >>> A1: Demote the API in the major version bump and remove the > > code > > > > > >> with a > > > > > >>> minor version bump. > > > > > >>> A2: Do exactly the same as 1-A, except that when removing > the > > > > code, > > > > > >>> bump up the major version. So this might be a short major > > version. > > > > > >>> > > > > > >>> B. If the maintenance overhead of the deprecated API is not > high, > > > > > >> therefore > > > > > >>> keeping it for a long time is affordable. There are also two > > > options: > > > > > >>> B1: Same as A-1, demote the API in the major version bump > and > > > > remove > > > > > >>> the code with a minor version bump. > > > > > >>> B2: Keep the API for all the minor versions in the major > > > version, > > > > > and > > > > > >>> only remove the code in the next major version. > > > > > >>> > > > > > >>> For case B, do we agree that B2 is the way to go? > > > > > >>> > > > > > >>> For case A: > > > > > >>> The following stays the same for A1 and A2: > > > > > >>> - users will lose the API after two minor leases. So the > > migration > > > > > >> period > > > > > >>> is the same for A-1 and A-2. > > > > > >>> - A1 and A2 will have exactly the same code after the removal > of > > > > > >>> deprecated API, the only difference is versioning. > > > > > >>> - To move forward, users need to move to the next minor > version > > in > > > > A1, > > > > > >> or > > > > > >>> to the next major version in A2. Because the code are the same, > > the > > > > > >> actual > > > > > >>> effort to upgrade is the same for A1 and A2. > > > > > >>> > > > > > >>> The differences between A-1 and A-2 are: > > > > > >>> - A1 allows keeping the major version release cadence. A-2 > will > > > > have a > > > > > >>> short major version release. > > > > > >>> - A1 breaks the well understood API semantic, while A-2 does > > not. > > > > > >>> > > > > > >>> From what I see, since there is no well establish standard > > > regarding > > > > > how > > > > > >>> long a major version should be released, A short major version > > > > release > > > > > is > > > > > >>> potential and emotional. It is not ideal but does not have > > material > > > > > >>> downsides compared with A1. I did not hear anyone complaining > > about > > > > > Kafka > > > > > >>> only has two 1.x release. However, A1 actually breaks the well > > > > > understood > > > > > >>> API semantic, which has more material impact. > > > > > >>> > > > > > >>> Also, I'd imagine 90% or more of the Public APIs should fall > into > > > > case > > > > > B. > > > > > >>> So, short major versions should be very occasional. I'd be very > > > > > concerned > > > > > >>> if the reason we choose A1 is simply because we cannot afford > > > > > >> maintaining a > > > > > >>> bunch of deprecated APIs until the next major version. This > > > indicates > > > > > >> that > > > > > >>> the actual problem we need to solve is to lower the maintenance > > > > > overhead > > > > > >> of > > > > > >>> deprecated APIs, so that we are comfortable to keep them > longer. > > As > > > > > John > > > > > >>> and I mentioned earlier, there are ways to achieve this and we > > need > > > > to > > > > > >>> learn how to do it in Flink. Otherwise, our discussion about > > > > versioning > > > > > >>> here does not bring much value, because we will end up with a > > bunch > > > > of > > > > > >>> short-lived APIs which upset our users, no matter how we > version > > > the > > > > > >>> releases. > > > > > >>> > > > > > >>> So, if there are concrete examples that you think will block us > > > from > > > > > >>> keeping API stability with affordable cost, let's take a look > > > > together > > > > > >> and > > > > > >>> see if that can be improved. > > > > > >>> > > > > > >>> Thanks, > > > > > >>> > > > > > >>> Jiangjie (Becket) Qin > > > > > >>> > > > > > >>> > > > > > >>> From what I see, > > > > > >>> > > > > > >>> > > > > > >>> On Mon, Jun 19, 2023 at 4:45 PM Xintong Song < > > > tonysong...@gmail.com > > > > > <mailto:tonysong...@gmail.com>> > > > > > >> wrote: > > > > > >>> > > > > > >>>>> > > > > > >>>>> The part I don't understand is if we are willing to have a > > > > migration > > > > > >>>>> period, and do a minor version bump to remove an API, what do > > we > > > > > >> lose to > > > > > >>>> do > > > > > >>>>> a major version bump instead, so we don't break the common > > > > versioning > > > > > >>>>> semantic? > > > > > >>>>> > > > > > >>>> > > > > > >>>> I think we are talking about the cases where a major version > > bump > > > > > >> happens > > > > > >>>> before a deprecated Public API reaches its migration period. > So > > > > > >> removing > > > > > >>>> the API with another major version bump means we have two > > > > consecutive > > > > > >> major > > > > > >>>> versions in a very short time. And as previously mentioned, > > having > > > > > >> frequent > > > > > >>>> major version bumps would weaken the value of the commitment > > > "Public > > > > > >> API > > > > > >>>> stay compatible within a major version". I think users also > have > > > > > >>>> expectations about how long a major version should live, which > > > > should > > > > > >> be at > > > > > >>>> least a couple of years rather than 1-2 minor releases. > > > > > >>>> > > > > > >>>> This is another option, but I think it is very likely more > > > expensive > > > > > >> than > > > > > >>>>> simply bumping the major version. And I imagine there will be > > > > > >> questions > > > > > >>>>> like "why this feature is in 1.20, but does not exist in > 2.0"? > > > > > >>>>> > > > > > >>>> > > > > > >>>> Yes, it's more expensive than simply bumping the major > version, > > > but > > > > > >> IMHO > > > > > >>>> it's probably cheaper than carrying the API until the next > major > > > > > >> version if > > > > > >>>> we don't bump the major version very soon. And see my reply > > above > > > on > > > > > >> why > > > > > >>>> not bumping immediately. > > > > > >>>> > > > > > >>>> Best, > > > > > >>>> > > > > > >>>> Xintong > > > > > >>>> > > > > > >>>> > > > > > >>>> > > > > > >>>> On Mon, Jun 19, 2023 at 4:22 PM Becket Qin < > > becket....@gmail.com > > > > > <mailto:becket....@gmail.com>> > > > > > >> wrote: > > > > > >>>> > > > > > >>>>> Hi Xintong, > > > > > >>>>> > > > > > >>>>> Please see the replies below. > > > > > >>>>> > > > > > >>>>> I see your point, that users would feel surprised if they > find > > > > > >> things no > > > > > >>>>>> longer work when upgrading to another 2.x minor release. > > > However, > > > > > >> I'd > > > > > >>>>> like > > > > > >>>>>> to point out that PublicEvolving APIs would have the similar > > > > > >> problem > > > > > >>>>>> anyway. So the question is, how do we catch users' attention > > and > > > > > >> make > > > > > >>>>> sure > > > > > >>>>>> they are aware that the Public APIs in 1.x may no longer be > > > Public > > > > > >> in > > > > > >>>>> 2.0. > > > > > >>>>>> There're various ways to do that, e.g., release notes, > > warnings > > > in > > > > > >>>> logs, > > > > > >>>>>> etc. > > > > > >>>>> > > > > > >>>>> First of all, I am not a fan of removing PublicEvolving APIs > in > > > > minor > > > > > >>>>> version changes. Personally speaking, I want them to be also > > > > removed > > > > > >> in > > > > > >>>>> major version changes. > > > > > >>>>> > > > > > >>>>> Empirically, I rarely see projects with complex API semantic > > work > > > > > >> well. > > > > > >>>>> Many, if not most, users don't read docs / release notes / > > > warnings > > > > > >> in > > > > > >>>>> logs. I don't think that is their fault. Typically an > > application > > > > > >>>> developer > > > > > >>>>> will have to deal with dozens of libraries maintained by > > various > > > > > >>>>> communities / groups. It is just too difficult for them to > keep > > > > > >> track of > > > > > >>>>> all the specific semantics each project puts there. If you > > think > > > > > >> about > > > > > >>>> it, > > > > > >>>>> how many of Flink developers read all the release notes of > > Guava, > > > > > >> Apache > > > > > >>>>> Commons, Apache Avro, ProtoBuf, etc, when you upgrade a > version > > > of > > > > > >> them? > > > > > >>>>> One would probably try bumping the dependency version and see > > if > > > > > >> there is > > > > > >>>>> an exception and solve them case by case. And if it is a > minor > > > > > >> version > > > > > >>>>> bump, one probably does not expect an exception at all. > Another > > > > > >> example > > > > > >>>> is > > > > > >>>>> that in Flink we still have so many usage of deprecated > methods > > > all > > > > > >> over > > > > > >>>>> the place, and I strongly doubt everyone knows when the > source > > > code > > > > > >> of > > > > > >>>>> these methods are deprecated and when they should be removed. > > > > > >>>>> > > > > > >>>>> So, the most important principle of API is simple and > > intuitive. > > > > The > > > > > >>>>> versioning semantic is a simple and universally accepted API > > > > > >> stability > > > > > >>>>> standard. If we ourselves as Flink developers are relying on > > this > > > > > >> for our > > > > > >>>>> own dependencies. I don't think we can expect more from our > > > users. > > > > > >>>>> > > > > > >>>>> Another possible alternative: whenever there's a deprecated > > > Public > > > > > >> API > > > > > >>>> that > > > > > >>>>>> reaches a major version bump before the migration period, > and > > we > > > > > >> also > > > > > >>>>> don't > > > > > >>>>>> want to carry it for all the next major release series, we > may > > > > > >> consider > > > > > >>>>>> releasing more minor releases for the previous major version > > > after > > > > > >> the > > > > > >>>>>> bump. E.g., an Public API is deprecated in 1.19, and then we > > > bump > > > > > >> to > > > > > >>>> 2.0, > > > > > >>>>>> we can release one more 1.20 after 2.0. That should provide > > > users > > > > > >>>> another > > > > > >>>>>> choice rather than upgrading to 2.0, while satisfying the > > > > > >>>> 2-minor-release > > > > > >>>>>> migration period. > > > > > >>>>> > > > > > >>>>> This is another option, but I think it is very likely more > > > > expensive > > > > > >> than > > > > > >>>>> simply bumping the major version. And I imagine there will be > > > > > >> questions > > > > > >>>>> like "why this feature is in 1.20, but does not exist in > 2.0"? > > > > > >>>>> > > > > > >>>>> I think my major point is, we should not carry APIs > deprecated > > > in a > > > > > >>>>>> previous major version along all the next major version > > series. > > > > I'd > > > > > >>>> like > > > > > >>>>> to > > > > > >>>>>> try giving users more commitments, i.e. the migration > period, > > as > > > > > >> long > > > > > >>>> as > > > > > >>>>> it > > > > > >>>>>> does not prevent us from making breaking changes. If it > > doesn't > > > > > >> work, > > > > > >>>> I'd > > > > > >>>>>> be in favor of not providing the migration period, but > > fallback > > > to > > > > > >> only > > > > > >>>>>> guarantee the compatibility within the major version. > > > > > >>>>> > > > > > >>>>> The part I don't understand is if we are willing to have a > > > > migration > > > > > >>>>> period, and do a minor version bump to remove an API, what do > > we > > > > > >> lose to > > > > > >>>> do > > > > > >>>>> a major version bump instead, so we don't break the common > > > > versioning > > > > > >>>>> semantic? > > > > > >>>>> > > > > > >>>>> Thanks, > > > > > >>>>> > > > > > >>>>> Jiangjie (Becket) Qin > > > > > >>>>> > > > > > >>>>> > > > > > >>>>> On Mon, Jun 19, 2023 at 3:20 PM Xintong Song < > > > > tonysong...@gmail.com > > > > > <mailto:tonysong...@gmail.com>> > > > > > >>>>> wrote: > > > > > >>>>> > > > > > >>>>>>> > > > > > >>>>>>> As an end user who only uses Public APIs, if I don't change > > my > > > > > >> code > > > > > >>>> at > > > > > >>>>>>> all, my expectation is the following: > > > > > >>>>>>> 1. Upgrading from 1.x to 2.x may have issues. > > > > > >>>>>>> 2. If I can upgrade from 1.x to 2.x without an issue, I am > > fine > > > > > >> with > > > > > >>>>> all > > > > > >>>>>>> the 2.x versions. > > > > > >>>>>>> Actually I think there are some dependency version > resolution > > > > > >>>> policies > > > > > >>>>>> out > > > > > >>>>>>> there which picks the highest minor version when the > > > dependencies > > > > > >>>> pull > > > > > >>>>> in > > > > > >>>>>>> multiple minor versions of the same jar, which may be > broken > > if > > > > > >> we > > > > > >>>>> remove > > > > > >>>>>>> the API in minor releases. > > > > > >>>>>>> > > > > > >>>>>> > > > > > >>>>>> I see your point, that users would feel surprised if they > find > > > > > >> things > > > > > >>>> no > > > > > >>>>>> longer work when upgrading to another 2.x minor release. > > > However, > > > > > >> I'd > > > > > >>>>> like > > > > > >>>>>> to point out that PublicEvolving APIs would have the similar > > > > > >> problem > > > > > >>>>>> anyway. So the question is, how do we catch users' attention > > and > > > > > >> make > > > > > >>>>> sure > > > > > >>>>>> they are aware that the Public APIs in 1.x may no longer be > > > Public > > > > > >> in > > > > > >>>>> 2.0. > > > > > >>>>>> There're various ways to do that, e.g., release notes, > > warnings > > > in > > > > > >>>> logs, > > > > > >>>>>> etc. > > > > > >>>>>> > > > > > >>>>>> Another possible alternative: whenever there's a deprecated > > > Public > > > > > >> API > > > > > >>>>> that > > > > > >>>>>> reaches a major version bump before the migration period, > and > > we > > > > > >> also > > > > > >>>>> don't > > > > > >>>>>> want to carry it for all the next major release series, we > may > > > > > >> consider > > > > > >>>>>> releasing more minor releases for the previous major version > > > after > > > > > >> the > > > > > >>>>>> bump. E.g., an Public API is deprecated in 1.19, and then we > > > bump > > > > > >> to > > > > > >>>> 2.0, > > > > > >>>>>> we can release one more 1.20 after 2.0. That should provide > > > users > > > > > >>>> another > > > > > >>>>>> choice rather than upgrading to 2.0, while satisfying the > > > > > >>>> 2-minor-release > > > > > >>>>>> migration period. > > > > > >>>>>> > > > > > >>>>>> I think my major point is, we should not carry APIs > deprecated > > > in > > > > a > > > > > >>>>>> previous major version along all the next major version > > series. > > > > I'd > > > > > >>>> like > > > > > >>>>> to > > > > > >>>>>> try giving users more commitments, i.e. the migration > period, > > as > > > > > >> long > > > > > >>>> as > > > > > >>>>> it > > > > > >>>>>> does not prevent us from making breaking changes. If it > > doesn't > > > > > >> work, > > > > > >>>> I'd > > > > > >>>>>> be in favor of not providing the migration period, but > > fallback > > > to > > > > > >> only > > > > > >>>>>> guarantee the compatibility within the major version. > > > > > >>>>>> > > > > > >>>>>> Best, > > > > > >>>>>> > > > > > >>>>>> Xintong > > > > > >>>>>> > > > > > >>>>>> > > > > > >>>>>> > > > > > >>>>>> On Mon, Jun 19, 2023 at 10:48 AM John Roesler < > > > > vvcep...@apache.org > > > > > <mailto:vvcep...@apache.org> > > > > > >>> > > > > > >>>>> wrote: > > > > > >>>>>> > > > > > >>>>>>> Hi Becket, > > > > > >>>>>>> > > > > > >>>>>>> Thanks for the reply! I’d like to continue the conversation > > > about > > > > > >>>>>>> compatibility outside of this FLIP thread, but for now, I > can > > > > > >> accept > > > > > >>>>> your > > > > > >>>>>>> decision. It’s certainly an improvement. > > > > > >>>>>>> > > > > > >>>>>>> Thanks again, > > > > > >>>>>>> John > > > > > >>>>>>> > > > > > >>>>>>> On Sun, Jun 18, 2023, at 21:42, Becket Qin wrote: > > > > > >>>>>>>> Hi John, > > > > > >>>>>>>> > > > > > >>>>>>>> Completely agree with all you said. > > > > > >>>>>>>> > > > > > >>>>>>>> Can we consider only dropping deprecated APIs in major > > > releases > > > > > >>>>> across > > > > > >>>>>>> the > > > > > >>>>>>>>> board? I understand that Experimental and PublicEvolving > > APIs > > > > > >> are > > > > > >>>> by > > > > > >>>>>>>>> definition less stable, but it seems like this should be > > > > > >> reflected > > > > > >>>>> in > > > > > >>>>>>> the > > > > > >>>>>>>>> required deprecation period alone. I.e. that we must keep > > > them > > > > > >>>>> around > > > > > >>>>>>> for > > > > > >>>>>>>>> at least zero or one minor release, not that we can drop > > them > > > > > >> in a > > > > > >>>>>>> minor or > > > > > >>>>>>>>> patch release. > > > > > >>>>>>>> > > > > > >>>>>>>> Personally speaking, I would love to do this, for exactly > > the > > > > > >>>> reason > > > > > >>>>>> you > > > > > >>>>>>>> mentioned. However, I did not propose this due to the > > > following > > > > > >>>>>> reasons: > > > > > >>>>>>>> > > > > > >>>>>>>> 1. I am hesitating a little bit about changing the > accepted > > > > > >> FLIPs > > > > > >>>> too > > > > > >>>>>>> soon. > > > > > >>>>>>>> 2. More importantly, to avoid slowing down our > development. > > At > > > > > >> this > > > > > >>>>>>> point, > > > > > >>>>>>>> Flink still lacks some design / routines to support good > API > > > > > >>>>>>> evolvability / > > > > > >>>>>>>> extensibility. Just like you said, it takes some time to > be > > > > > >> good at > > > > > >>>>>> this. > > > > > >>>>>>>> In this case, my concern is that only removing > Experimental > > / > > > > > >>>>>>>> PublicEvolving APIs in major version changes may result in > > too > > > > > >> much > > > > > >>>>>>>> overhead and dramatically slow down the development of > > Flink. > > > > > >> So, I > > > > > >>>>> was > > > > > >>>>>>>> thinking that we can start with the current status. > > Hopefully > > > > > >> after > > > > > >>>>> we > > > > > >>>>>>> are > > > > > >>>>>>>> more comfortable with the maintenance overhead of > deprecated > > > > > >> APIs, > > > > > >>>> we > > > > > >>>>>> can > > > > > >>>>>>>> then have a stronger guarantee for Experimental / > > > > > >> PublicEvolving > > > > > >>>>> APIs. > > > > > >>>>>>>> > > > > > >>>>>>>> Thanks, > > > > > >>>>>>>> > > > > > >>>>>>>> Jiangjie (Becket) Qin > > > > > >>>>>>>> > > > > > >>>>>>>> > > > > > >>>>>>>> > > > > > >>>>>>>> On Sun, Jun 18, 2023 at 6:44 AM John Roesler < > > > > > >> vvcep...@apache.org <mailto:vvcep...@apache.org>> > > > > > >>>>>>> wrote: > > > > > >>>>>>>> > > > > > >>>>>>>>> Hi Becket, > > > > > >>>>>>>>> > > > > > >>>>>>>>> Thanks for this FLIP! Having a deprecation process is > > really > > > > > >>>>>> important. > > > > > >>>>>>> I > > > > > >>>>>>>>> understand some people’s concerns about the additional > > burden > > > > > >> for > > > > > >>>>>>> project > > > > > >>>>>>>>> maintainers, but my personal experience with Kafka has > been > > > > > >> that > > > > > >>>>> it’s > > > > > >>>>>>> very > > > > > >>>>>>>>> liveable and that it’s well worth the benefit to users. > In > > > > > >> fact, > > > > > >>>>> users > > > > > >>>>>>>>> being able to confidently upgrade is also a benefit to > > > > > >>>> maintainers, > > > > > >>>>> as > > > > > >>>>>>> we > > > > > >>>>>>>>> will get fewer questions from people stuck on very old > > > > > >> versions. > > > > > >>>>>>>>> > > > > > >>>>>>>>> One question: > > > > > >>>>>>>>> Can we consider only dropping deprecated APIs in major > > > > > >> releases > > > > > >>>>> across > > > > > >>>>>>> the > > > > > >>>>>>>>> board? I understand that Experimental and PublicEvolving > > APIs > > > > > >> are > > > > > >>>> by > > > > > >>>>>>>>> definition less stable, but it seems like this should be > > > > > >> reflected > > > > > >>>>> in > > > > > >>>>>>> the > > > > > >>>>>>>>> required deprecation period alone. I.e. that we must keep > > > them > > > > > >>>>> around > > > > > >>>>>>> for > > > > > >>>>>>>>> at least zero or one minor release, not that we can drop > > them > > > > > >> in a > > > > > >>>>>>> minor or > > > > > >>>>>>>>> patch release. > > > > > >>>>>>>>> > > > > > >>>>>>>>> The advantage of forbidding the removal of any API in > minor > > > or > > > > > >>>> patch > > > > > >>>>>>>>> releases is that users will get a strong guarantee that > > they > > > > > >> can > > > > > >>>>> bump > > > > > >>>>>>> the > > > > > >>>>>>>>> minor or patch version and still be able to compile, or > > even > > > > > >> just > > > > > >>>>>>> re-link > > > > > >>>>>>>>> and know that they won’t face “MethodDef” exceptions at > run > > > > > >> time. > > > > > >>>>> This > > > > > >>>>>>> is a > > > > > >>>>>>>>> binary guarantee: if we allow removing even Experimental > > > APIs > > > > > >>>>> outside > > > > > >>>>>>> of > > > > > >>>>>>>>> major releases, users can no longer confidently upgrade. > > > > > >>>>>>>>> > > > > > >>>>>>>>> Aside from that, I’d share my 2 cents on a couple of > > points: > > > > > >>>>>>>>> * I’d use the official Deprecated annotation instead of > > > > > >>>> introducing > > > > > >>>>>> our > > > > > >>>>>>>>> own flavor (Retired, etc), since Deprecated is well > > > integrated > > > > > >>>> into > > > > > >>>>>>> build > > > > > >>>>>>>>> tools and IDEs. > > > > > >>>>>>>>> * I wouldn’t worry about a demotion process in this FLIP; > > it > > > > > >> seems > > > > > >>>>>>>>> orthogonal, and something that should probably be taken > > > > > >>>> case-by-case > > > > > >>>>>>>>> anyway. > > > > > >>>>>>>>> * Aside from deprecation and removal, there have been > some > > > > > >>>>> discussions > > > > > >>>>>>>>> about how to evolve APIs and behavior in compatible ways. > > > > > >> This is > > > > > >>>>>>> somewhat > > > > > >>>>>>>>> of an art, and if folks haven’t wrestled with it before, > > > it’ll > > > > > >>>> take > > > > > >>>>>> some > > > > > >>>>>>>>> time to become good at it. I feel like this topic should > > also > > > > > >> be > > > > > >>>>>>> orthogonal > > > > > >>>>>>>>> to this FLIP, but FWIW, my suggestion would be to adopt a > > > > > >> simple > > > > > >>>>>> policy > > > > > >>>>>>> not > > > > > >>>>>>>>> to break existing user programs, and leave the “how” up > to > > > > > >>>>>> implementers > > > > > >>>>>>> and > > > > > >>>>>>>>> reviewers. > > > > > >>>>>>>>> > > > > > >>>>>>>>> Thanks again, > > > > > >>>>>>>>> John > > > > > >>>>>>>>> > > > > > >>>>>>>>> On Sat, Jun 17, 2023, at 11:03, Jing Ge wrote: > > > > > >>>>>>>>>> Hi All, > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> The @Public -> @PublicEvolving proposed by Xintong is a > > > > > >> great > > > > > >>>>> idea. > > > > > >>>>>>>>>> Especially, after he suggest @PublicRetired, i.e. > > > > > >>>> @PublicEvolving > > > > > >>>>>> --(2 > > > > > >>>>>>>>>> minor release)--> @Public --> @deprecated --(1 major > > > > > >>>>>>>>>> release)--> @PublicRetired. It will provide a lot of > > > > > >> flexibility > > > > > >>>>>>> without > > > > > >>>>>>>>>> breaking any rules we had. @Public APIs are allowed to > > > > > >> change > > > > > >>>>>> between > > > > > >>>>>>>>> major > > > > > >>>>>>>>>> releases. Changing annotations is acceptable and > provides > > > > > >>>>> additional > > > > > >>>>>>>>>> tolerance i.e. user-friendliness, since the APIs > themself > > > > > >> are > > > > > >>>> not > > > > > >>>>>>>>> changed. > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> I had similar thoughts when I was facing those issues. I > > > > > >> want to > > > > > >>>>>> move > > > > > >>>>>>> one > > > > > >>>>>>>>>> step further and suggest introducing one more annotation > > > > > >>>> @Retired. > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> Not like the @PublicRetired which is a compromise of > > > > > >> downgrading > > > > > >>>>>>> @Public > > > > > >>>>>>>>> to > > > > > >>>>>>>>>> @PublicEvolving. As I mentioned earlier in my reply, > Java > > > > > >>>> standard > > > > > >>>>>>>>>> @deprecated should be used in the early stage of the > > > > > >> deprecation > > > > > >>>>>>> process > > > > > >>>>>>>>>> and doesn't really meet our requirement. Since Java does > > not > > > > > >>>> allow > > > > > >>>>>> us > > > > > >>>>>>> to > > > > > >>>>>>>>>> extend annotation, I think it would be feasible to have > > the > > > > > >> new > > > > > >>>>>>> @Retired > > > > > >>>>>>>>> to > > > > > >>>>>>>>>> help us monitor and manage the deprecation process, > house > > > > > >>>>> cleaning, > > > > > >>>>>>> etc. > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> Some ideas could be(open for discussion): > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> @Retired: > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> 1. There must be a replacement with functionality > > > > > >> compatibility > > > > > >>>>>> before > > > > > >>>>>>>>> APIs > > > > > >>>>>>>>>> can be marked as @Retired, i.e. DISCUSS and VOTE > processes > > > > > >> on > > > > > >>>> the > > > > > >>>>> ML > > > > > >>>>>>> are > > > > > >>>>>>>>>> mandatory (a FLIP is recommended). > > > > > >>>>>>>>>> 2. APIs marked as @Retired will be removed after 1 minor > > > > > >> release > > > > > >>>>>>> sharply > > > > > >>>>>>>>>> (using ArchUnit to force it, needs further check whether > > it > > > > > >> is > > > > > >>>>>>> possible). > > > > > >>>>>>>>>> Devs who marked them as @Retired are responsible to > remove > > > > > >> them. > > > > > >>>>>>>>>> 3. Both @Public -> @Retired and @PublicEvolving -> > > @Retired > > > > > >> are > > > > > >>>>>>>>>> recommended. @Experimental -> @Retired and @Internal -> > > > > > >> @Retired > > > > > >>>>>> could > > > > > >>>>>>>>> also > > > > > >>>>>>>>>> be used if it can increase user-friendliness or > > > > > >>>> dev-friendliness, > > > > > >>>>>> but > > > > > >>>>>>> not > > > > > >>>>>>>>>> mandatory. > > > > > >>>>>>>>>> 4. Some variables will be defined in @Retired to support > > the > > > > > >>>>>>> deprecation > > > > > >>>>>>>>>> process management. Further extension is possible, since > > the > > > > > >>>>>>> annotation > > > > > >>>>>>>>> is > > > > > >>>>>>>>>> built by us. > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> Best regards, > > > > > >>>>>>>>>> Jing > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> On Fri, Jun 16, 2023 at 10:31 AM Becket Qin < > > > > > >>>> becket....@gmail.com <mailto:becket....@gmail.com> > > > > > >>>>>> > > > > > >>>>>>>>> wrote: > > > > > >>>>>>>>>> > > > > > >>>>>>>>>>> Hi Xintong, > > > > > >>>>>>>>>>> > > > > > >>>>>>>>>>> Thanks for the explanation. Please see the replies > inline > > > > > >>>> below. > > > > > >>>>>>>>>>> > > > > > >>>>>>>>>>> I agree. And from my understanding, demoting a Public > API > > > > > >> is > > > > > >>>>> also a > > > > > >>>>>>>>> kind of > > > > > >>>>>>>>>>>> such change, just like removing one, which can only > > > > > >> happen > > > > > >>>> with > > > > > >>>>>>> major > > > > > >>>>>>>>>>>> version bumps. I'm not proposing to allow demoting > > Public > > > > > >>>> APIs > > > > > >>>>>>>>> anytime, > > > > > >>>>>>>>>>> but > > > > > >>>>>>>>>>>> only in the case major version bumps happen before > > > > > >> reaching > > > > > >>>> the > > > > > >>>>>>>>>>>> 2-minor-release migration period. Actually, demoting > > > > > >> would > > > > > >>>> be a > > > > > >>>>>>> weaker > > > > > >>>>>>>>>>>> change compared to removing the API immediately upon > > > > > >> major > > > > > >>>>>> version > > > > > >>>>>>>>> bumps, > > > > > >>>>>>>>>>>> in order to keep the commitment about the > > 2-minor-release > > > > > >>>>>> migration > > > > > >>>>>>>>>>> period. > > > > > >>>>>>>>>>>> If the concern is that `@Public` -> `@PublicEvolving` > > > > > >> sounds > > > > > >>>>>>> against > > > > > >>>>>>>>>>>> conventions, we may introduce a new annotation if > > > > > >> necessary, > > > > > >>>>>> e.g., > > > > > >>>>>>>>>>>> `@PublicRetiring`, to avoid confusions. > > > > > >>>>>>>>>>> > > > > > >>>>>>>>>>> As an end user who only uses Public APIs, if I don't > > > > > >> change my > > > > > >>>>> code > > > > > >>>>>>> at > > > > > >>>>>>>>> all, > > > > > >>>>>>>>>>> my expectation is the following: > > > > > >>>>>>>>>>> 1. Upgrading from 1.x to 2.x may have issues. > > > > > >>>>>>>>>>> 2. If I can upgrade from 1.x to 2.x without an issue, I > > am > > > > > >> fine > > > > > >>>>>> with > > > > > >>>>>>> all > > > > > >>>>>>>>>>> the 2.x versions. > > > > > >>>>>>>>>>> Actually I think there are some dependency version > > > > > >> resolution > > > > > >>>>>>> policies > > > > > >>>>>>>>> out > > > > > >>>>>>>>>>> there which picks the highest minor version when the > > > > > >>>> dependencies > > > > > >>>>>>> pull > > > > > >>>>>>>>> in > > > > > >>>>>>>>>>> multiple minor versions of the same jar, which may be > > > > > >> broken if > > > > > >>>>> we > > > > > >>>>>>>>> remove > > > > > >>>>>>>>>>> the API in minor releases. > > > > > >>>>>>>>>>> > > > > > >>>>>>>>>>> I'm not sure about this. Yes, it's completely "legal" > > that > > > > > >> we > > > > > >>>>> bump > > > > > >>>>>> up > > > > > >>>>>>>>> the > > > > > >>>>>>>>>>>> major version whenever a breaking change is needed. > > > > > >> However, > > > > > >>>>> this > > > > > >>>>>>> also > > > > > >>>>>>>>>>>> weakens the value of the commitment that public APIs > > will > > > > > >>>> stay > > > > > >>>>>>> stable > > > > > >>>>>>>>>>>> within the major release series, as the series can end > > > > > >>>> anytime. > > > > > >>>>>>> IMHO, > > > > > >>>>>>>>>>> short > > > > > >>>>>>>>>>>> major release series are not something "make the end > > > > > >> users > > > > > >>>>>> happy", > > > > > >>>>>>> but > > > > > >>>>>>>>>>>> backdoors that allow us as the developers to make > > > > > >> frequent > > > > > >>>>>> breaking > > > > > >>>>>>>>>>>> changes. On the contrary, with the demoting approach, > we > > > > > >> can > > > > > >>>>>> still > > > > > >>>>>>>>> have > > > > > >>>>>>>>>>>> longer major release series, while only allowing > Public > > > > > >> APIs > > > > > >>>>>>>>> deprecated > > > > > >>>>>>>>>>> at > > > > > >>>>>>>>>>>> the end of the previous major version to be removed in > > > > > >> the > > > > > >>>> next > > > > > >>>>>>> major > > > > > >>>>>>>>>>>> version. > > > > > >>>>>>>>>>> > > > > > >>>>>>>>>>> I totally agree that frequent major version bumps are > not > > > > > >>>> ideal, > > > > > >>>>>> but > > > > > >>>>>>>>> here > > > > > >>>>>>>>>>> we are comparing it with a minor version bump which > > > > > >> removes a > > > > > >>>>>> Public > > > > > >>>>>>>>> API. > > > > > >>>>>>>>>>> So the context is that we have already decided to > remove > > > > > >> this > > > > > >>>>>> Public > > > > > >>>>>>> API > > > > > >>>>>>>>>>> while keeping everything else backwards compatible. I > > > > > >> think a > > > > > >>>>> major > > > > > >>>>>>>>> version > > > > > >>>>>>>>>>> bump is a commonly understood signal here, compared > with > > a > > > > > >>>> minor > > > > > >>>>>>> version > > > > > >>>>>>>>>>> change. From end users' perspective, for those who are > > not > > > > > >>>>>> impacted, > > > > > >>>>>>> in > > > > > >>>>>>>>>>> this case upgrading a major version is not necessarily > > more > > > > > >>>>>> involved > > > > > >>>>>>>>> than > > > > > >>>>>>>>>>> upgrading a minor version - both should be as smooth > as a > > > > > >>>>>> dependency > > > > > >>>>>>>>>>> version change. For those who are impacted, they will > > lose > > > > > >> the > > > > > >>>>>> Public > > > > > >>>>>>>>> API > > > > > >>>>>>>>>>> anyways and a major version bump ensures there is no > > > > > >> surprise. > > > > > >>>>>>>>>>> > > > > > >>>>>>>>>>> Thanks, > > > > > >>>>>>>>>>> > > > > > >>>>>>>>>>> Jiangjie (Becket) Qin > > > > > >>>>>>>>>>> > > > > > >>>>>>>>>>> On Fri, Jun 16, 2023 at 10:13 AM Xintong Song < > > > > > >>>>>> tonysong...@gmail.com <mailto:tonysong...@gmail.com> > > > > > >>>>>>>> > > > > > >>>>>>>>>>> wrote: > > > > > >>>>>>>>>>> > > > > > >>>>>>>>>>>> Public API is a well defined common concept, and one > of > > > > > >> its > > > > > >>>>>>>>>>>>> convention is that it only changes with a major > version > > > > > >>>>> change. > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>> > > > > > >>>>>>>>>>>> I agree. And from my understanding, demoting a Public > > > > > >> API is > > > > > >>>>>> also a > > > > > >>>>>>>>> kind > > > > > >>>>>>>>>>>> of such change, just like removing one, which can only > > > > > >> happen > > > > > >>>>>> with > > > > > >>>>>>>>> major > > > > > >>>>>>>>>>>> version bumps. I'm not proposing to allow demoting > > Public > > > > > >>>> APIs > > > > > >>>>>>>>> anytime, > > > > > >>>>>>>>>>> but > > > > > >>>>>>>>>>>> only in the case major version bumps happen before > > > > > >> reaching > > > > > >>>> the > > > > > >>>>>>>>>>>> 2-minor-release migration period. Actually, demoting > > > > > >> would > > > > > >>>> be a > > > > > >>>>>>> weaker > > > > > >>>>>>>>>>>> change compared to removing the API immediately upon > > > > > >> major > > > > > >>>>>> version > > > > > >>>>>>>>> bumps, > > > > > >>>>>>>>>>>> in order to keep the commitment about the > > 2-minor-release > > > > > >>>>>> migration > > > > > >>>>>>>>>>> period. > > > > > >>>>>>>>>>>> If the concern is that `@Public` -> `@PublicEvolving` > > > > > >> sounds > > > > > >>>>>>> against > > > > > >>>>>>>>>>>> conventions, we may introduce a new annotation if > > > > > >> necessary, > > > > > >>>>>> e.g., > > > > > >>>>>>>>>>>> `@PublicRetiring`, to avoid confusions. > > > > > >>>>>>>>>>>> > > > > > >>>>>>>>>>>> But it should be > > > > > >>>>>>>>>>>>> completely OK to bump up the major version if we > really > > > > > >> want > > > > > >>>>> to > > > > > >>>>>>> get > > > > > >>>>>>>>> rid > > > > > >>>>>>>>>>> of > > > > > >>>>>>>>>>>>> a public API, right? > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>> > > > > > >>>>>>>>>>>> I'm not sure about this. Yes, it's completely "legal" > > > > > >> that we > > > > > >>>>>> bump > > > > > >>>>>>> up > > > > > >>>>>>>>> the > > > > > >>>>>>>>>>>> major version whenever a breaking change is needed. > > > > > >> However, > > > > > >>>>> this > > > > > >>>>>>> also > > > > > >>>>>>>>>>>> weakens the value of the commitment that public APIs > > will > > > > > >>>> stay > > > > > >>>>>>> stable > > > > > >>>>>>>>>>>> within the major release series, as the series can end > > > > > >>>> anytime. > > > > > >>>>>>> IMHO, > > > > > >>>>>>>>>>> short > > > > > >>>>>>>>>>>> major release series are not something "make the end > > > > > >> users > > > > > >>>>>> happy", > > > > > >>>>>>> but > > > > > >>>>>>>>>>>> backdoors that allow us as the developers to make > > > > > >> frequent > > > > > >>>>>> breaking > > > > > >>>>>>>>>>>> changes. On the contrary, with the demoting approach, > we > > > > > >> can > > > > > >>>>>> still > > > > > >>>>>>>>> have > > > > > >>>>>>>>>>>> longer major release series, while only allowing > Public > > > > > >> APIs > > > > > >>>>>>>>> deprecated > > > > > >>>>>>>>>>> at > > > > > >>>>>>>>>>>> the end of the previous major version to be removed in > > > > > >> the > > > > > >>>> next > > > > > >>>>>>> major > > > > > >>>>>>>>>>>> version. > > > > > >>>>>>>>>>>> > > > > > >>>>>>>>>>>> Given our track record I would prefer a regular cycle > > > > > >> (1-2 > > > > > >>>>> years) > > > > > >>>>>>> to > > > > > >>>>>>>>>>>>> force us to think about this whole topic, and not put > > it > > > > > >>>> again > > > > > >>>>>> to > > > > > >>>>>>> the > > > > > >>>>>>>>>>>>> wayside and giving us (and users) a clear expectation > > on > > > > > >>>> when > > > > > >>>>>>>>> breaking > > > > > >>>>>>>>>>>>> changes can be made. > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>> > > > > > >>>>>>>>>>>> +1. I personally think 2-3 years would be a good time > > > > > >> for new > > > > > >>>>>> major > > > > > >>>>>>>>>>>> versions, or longer if there's no breaking changes > > > > > >> needed. > > > > > >>>> That > > > > > >>>>>>> makes > > > > > >>>>>>>>> 1-2 > > > > > >>>>>>>>>>>> year a perfect time to revisit the topic, while > leaving > > > > > >> us > > > > > >>>> more > > > > > >>>>>>> time > > > > > >>>>>>>>> to > > > > > >>>>>>>>>>>> prepare the major release if needed. > > > > > >>>>>>>>>>>> > > > > > >>>>>>>>>>>> Best, > > > > > >>>>>>>>>>>> > > > > > >>>>>>>>>>>> Xintong > > > > > >>>>>>>>>>>> > > > > > >>>>>>>>>>>> > > > > > >>>>>>>>>>>> > > > > > >>>>>>>>>>>> On Thu, Jun 15, 2023 at 10:09 PM Chesnay Schepler < > > > > > >>>>>>> ches...@apache.org <mailto:ches...@apache.org> > > > > > >>>>>>>>>> > > > > > >>>>>>>>>>>> wrote: > > > > > >>>>>>>>>>>> > > > > > >>>>>>>>>>>>> On 13/06/2023 17:26, Becket Qin wrote: > > > > > >>>>>>>>>>>>>> It would be valuable if we can avoid releasing minor > > > > > >>>>> versions > > > > > >>>>>>> for > > > > > >>>>>>>>>>>>> previous > > > > > >>>>>>>>>>>>>> major versions. > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>>> On paper, /absolutely /agree, but I'm not sure how > > > > > >> viable > > > > > >>>> that > > > > > >>>>>> is > > > > > >>>>>>> in > > > > > >>>>>>>>>>>>> practice. > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>>> On the current 2.0 agenda is potentially dropping > > > > > >> support > > > > > >>>> for > > > > > >>>>>> Java > > > > > >>>>>>>>> 8/11, > > > > > >>>>>>>>>>>>> which may very well be a problem for our current > users. > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>>> On 13/06/2023 17:26, Becket Qin wrote: > > > > > >>>>>>>>>>>>>> Thanks for the feedback and sorry for the confusion > > > > > >> about > > > > > >>>>>> Public > > > > > >>>>>>>>> API > > > > > >>>>>>>>>>>>>> deprecation. I just noticed that there was a mistake > > > > > >> in > > > > > >>>> the > > > > > >>>>>>> NOTES > > > > > >>>>>>>>> part > > > > > >>>>>>>>>>>>> for > > > > > >>>>>>>>>>>>>> Public API due to a copy-paste error... I just fixed > > > > > >> it. > > > > > >>>>>>>>>>>>> I'm very relieved to hear that. Glad to hear that we > > > > > >> are on > > > > > >>>>> the > > > > > >>>>>>> same > > > > > >>>>>>>>>>>>> page on that note. > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>>> On 15/06/2023 15:20, Becket Qin wrote: > > > > > >>>>>>>>>>>>>> But it should be > > > > > >>>>>>>>>>>>>> completely OK to bump up the major version if we > > > > > >> really > > > > > >>>> want > > > > > >>>>>> to > > > > > >>>>>>> get > > > > > >>>>>>>>>>> rid > > > > > >>>>>>>>>>>>> of > > > > > >>>>>>>>>>>>>> a public API, right? > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>>> Technically yes, but look at how long it took to get > us > > > > > >> to > > > > > >>>>> 2.0. > > > > > >>>>>> ;) > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>>> There's a separate discussion to be had on the > cadence > > > > > >> of > > > > > >>>>> major > > > > > >>>>>>>>> releases > > > > > >>>>>>>>>>>>> going forward, and there seem to be different > opinions > > > > > >> on > > > > > >>>>> that. > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>>> If we take the Kafka example of 2 minor releases > > between > > > > > >>>> major > > > > > >>>>>>> ones, > > > > > >>>>>>>>>>>>> that for us means that users have to potentially deal > > > > > >> with > > > > > >>>>>>> breaking > > > > > >>>>>>>>>>>>> changes every 6 months, which seems like a lot. > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>>> Given our track record I would prefer a regular cycle > > > > > >> (1-2 > > > > > >>>>>> years) > > > > > >>>>>>> to > > > > > >>>>>>>>>>>>> force us to think about this whole topic, and not put > > it > > > > > >>>> again > > > > > >>>>>> to > > > > > >>>>>>> the > > > > > >>>>>>>>>>>>> wayside and giving us (and users) a clear expectation > > on > > > > > >>>> when > > > > > >>>>>>>>> breaking > > > > > >>>>>>>>>>>>> changes can be made. > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>>> But again, maybe this should be in a separate thread. > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>>> On 14/06/2023 11:37, Becket Qin wrote: > > > > > >>>>>>>>>>>>>> Do you have an example of behavioral change in mind? > > > > > >> Not > > > > > >>>>> sure > > > > > >>>>>> I > > > > > >>>>>>>>> fully > > > > > >>>>>>>>>>>>>> understand the concern for behavioral change here. > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>>> This could be a lot of things. It can be performance > in > > > > > >>>>> certain > > > > > >>>>>>>>>>>>> edge-cases, a bug fix that users (maybe unknowingly) > > > > > >> relied > > > > > >>>>> upon > > > > > >>>>>>>>>>>>> ( > > > > > > > > > > > > > > > https://www.google.com/url?q=https://xkcd.com/1172/&source=gmail-imap&ust=1687869831000000&usg=AOvVaw2QkbaBd4VKyCsxxFcv1S4I > > > > ), > > > > > a semantic change to some > > > > > >> API. > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>>> For a concrete example, consider the job submission. > A > > > > > >> few > > > > > >>>>>>> releases > > > > > >>>>>>>>> back > > > > > >>>>>>>>>>>>> we made changes such that the initialization of the > job > > > > > >>>> master > > > > > >>>>>>>>> happens > > > > > >>>>>>>>>>>>> asynchronously. > > > > > >>>>>>>>>>>>> This meant the job submission call returns sooner, > and > > > > > >> the > > > > > >>>> job > > > > > >>>>>>> state > > > > > >>>>>>>>>>>>> enum was extended to cover this state. > > > > > >>>>>>>>>>>>> API-wise we consider this a compatible change, but > the > > > > > >>>>> observed > > > > > >>>>>>>>> behavior > > > > > >>>>>>>>>>>>> may be different. > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>>> Metrics are another example; I believe over time we > > > > > >> changed > > > > > >>>>> what > > > > > >>>>>>> some > > > > > >>>>>>>>>>>>> metrics returned a few times. > > > > > > > > > > > > > > > > > > > >
