I believe it would be helpful to clearly define the following for our API management along with versioning:
1. *Valid APIs*: It's common to support multiple versions of a valid API. For example, both v1 and v2 of loadTable should remain functional. 2. *Deprecated APIs*: Even after deprecation, APIs should continue to be supported to avoid disruptions. 3. *Removing Deprecated APIs*: We need a careful approach for retiring deprecated APIs, as removing them can cause breaking changes for existing clients. This step should be treated as a breaking change and handled with caution. Additionally, it is helpful to differentiate between *specs* (e.g., REST endpoints) and *APIs* (public classes and methods). REST endpoints tend to have longer lifecycles and will be deprecated or removed according to a well-defined plan(discussion, announcement, etc), while public methods should adhere to semantic versioning. Yufei On Wed, Oct 23, 2024 at 4:55 AM Jean-Baptiste Onofré <j...@nanthrax.net> wrote: > By experience, it's very difficult to maintain two API versions in the > same codebase: very soon, we have a lot of "conditional statements" in > the code, which is not super elegant. Supporting previous API versions > (with routing) is possible as soon as we don't introduce "conditional > statements" per version. > The maintenance of branches is not necessarily painful: all new > features and improvements go on the latest active branch, other > branches just receive security fixes, dependency updates or major > fixes. > Else, we clearly state that we support only the latest active branch. > Several Apache projects maintain a couple of active branches (Apache > Camel, Apache ActiveMQ, etc). > > I think it makes sense to bump the Polaris version with the Iceberg > REST Spec version. The versions don't necessarily match (like Polaris > 3 can implement Iceberg REST Spec 2), but any new major REST Spec > version should result in a new major Polaris version. > > Regards > JB > > On Wed, Oct 23, 2024 at 1:05 PM Michael Collado <collado.m...@gmail.com> > wrote: > > > > Personally, I’d rather not maintain two branches. I think it’s easier to > > maintain two API versions in the same codebase rather than have to back > > port changes into stale branches. > > > > This also leads to the question of what we do for Iceberg spec API > version > > changes. I can imagine if the whole REST spec changes to v2, we might > > release Polaris 2.0, but what do we do if Iceberg only increments the > > version for one or two APIs? > > > > Mike > > > > On Tue, Oct 22, 2024 at 5:42 AM Jean-Baptiste Onofré <j...@nanthrax.net> > > wrote: > > > > > Good point. > > > > > > Imho, we should increment API versions when we introduce "breaking > > > changes". > > > On major versions, I don't see a need to support previous API > > > versions. Users who want to support previous API can always use > > > previous Polaris versions. > > > For us, we can "maintain" two major versions in parallel (the last one > > > with active new features, improvements, and bug fixes on both the last > > > version and previous one). > > > > > > For instance, in the Apache ActiveMQ project maintains both ActiveMQ > > > 5.x and 6.x, with complete different API support (ActiveMQ 5.x > > > implements javax.jms 1.1, whereas ActiveMQ 6.x implements jakarta.jms > > > 3.1). > > > > > > So, regarding your example, I would say that Polaris 2.0 should not > > > necessary support v1 APIs, but focus on vX APIs and important changes. > > > We can still maintain Polaris 1.x for v1 API (with dependency updates > > > and bug fixes). > > > > > > Regards > > > JB > > > > > > On Tue, Oct 22, 2024 at 12:10 AM Michael Collado < > collado.m...@gmail.com> > > > wrote: > > > > > > > > What's the support policy for API versions? Do we increment API > versions > > > > for breaking API changes (this seems reasonable and likely)? And do > we > > > > support previous API versions for some period of time? > > > > > > > > If the answer to both of those questions is "yes" , then what is the > > > > purpose of fast major verison releases? If we ship Polaris 2.0, > shouldn't > > > > the service still support the v1 APIs? For me, a 2.0 release should > mean > > > > pretty radical changes - new features, such as non-Iceberg table > support, > > > > delegated Authz policies, etc. > > > > > > > > On Mon, Oct 21, 2024 at 11:53 AM Yufei Gu <flyrain...@gmail.com> > wrote: > > > > > > > > > I'm not opposed to rapid evolution, but I’m trying to understand > the > > > > > reasoning behind the fast major version. It’s ultimately a tradeoff > > > between > > > > > maintaining backward compatibility and advancing quickly through > > > breaking > > > > > changes—we can’t achieve both at the same time. A faster major > version > > > > > release seems to indicate a preference for rapid progress over > > > stability. > > > > > Otherwise, why push for it? Am I missing something? > > > > > > > > > > Yufei > > > > > > > > > > > > > > > On Mon, Oct 21, 2024 at 11:26 AM Dmitri Bourlatchkov > > > > > <dmitri.bourlatch...@dremio.com.invalid> wrote: > > > > > > > > > > > IMHO, production compatibility concerns are valid, but they are > not > > > > > related > > > > > > to the versioning scheme. > > > > > > > > > > > > Backward-compatibility breakage should be declared anyway, > whether > > > the > > > > > move > > > > > > is from 0.42.1 to 0.43.0 or from 2.1.1 to 3.0.0. > > > > > > > > > > > > Still, bumping a major version indicates major change. It does > not > > > > > > automatically indicate incompatibility. My reading of Semver [1] > > > > > guidelines > > > > > > is that major must be incremented when incompatible changes are > > > present, > > > > > > but there is no restriction on incrementing major and keeping the > > > product > > > > > > backward compatible. > > > > > > > > > > > > On the other hand, if we avoid major code changes because of the > > > risk of > > > > > > incompatibility, that will slow down the project's evolution. > > > > > > > > > > > > Cheers, > > > > > > Dmitri. > > > > > > > > > > > > [1] https://semver.org/ > > > > > > > > > > > > On Mon, Oct 21, 2024 at 2:07 PM Yufei Gu <flyrain...@gmail.com> > > > wrote: > > > > > > > > > > > > > +1 on moving forward with 0.9 as the first version! > > > > > > > > > > > > > > I have some concerns about fast major versioning, especially > with > > > > > > > compatibility. Since the catalog is crucial in production, > teams > > > may be > > > > > > > reluctant to upgrade, particularly with breaking changes. > > > > > > > > > > > > > > Does fast versioning help in this case, or could it make things > > > harder? > > > > > > > Does it encourage more frequent upgrades, and is there a plan > to > > > manage > > > > > > > production impact? While we can't completely avoid user forks, > we > > > want > > > > > to > > > > > > > minimize divergence. > > > > > > > > > > > > > > Yufei > > > > > > > > > > > > > > > > > > > > > On Fri, Oct 18, 2024 at 6:33 AM Dmitri Bourlatchkov > > > > > > > <dmitri.bourlatch...@dremio.com.invalid> wrote: > > > > > > > > > > > > > > > +1 to "fast" major version. > > > > > > > > > > > > > > > > Cheers, > > > > > > > > Dmitri. > > > > > > > > > > > > > > > > On Fri, Oct 18, 2024 at 5:17 AM Jean-Baptiste Onofré < > > > > > j...@nanthrax.net> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi folks, > > > > > > > > > > > > > > > > > > First of all, thanks everyone for the first community > meeting > > > > > > > > > yesterday. It was a great meeting, very interesting > > > discussions. > > > > > > > > > You can find the meeting notes on the Polaris website > > > > > > > > > (https://polaris.apache.org/community/meetings/): > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1TAAMjCtk4KuWSwfxpCBhhK9vM1k_3n7YE4L28slclXU/edit?usp=sharing > > > > > > > > > > > > > > > > > > As discussed during the meeting, here's the release > proposal: > > > > > > > > > - the first Polaris release will be 0.9. I'm doing a new > pass > > > on > > > > > the > > > > > > > > > preparation steps (due diligence on > > > > > > > > > NOTICE/LICENSE/DISCLAIMER/dependencies, automatic > "scripts", > > > etc). > > > > > As > > > > > > > > > we are mid October, I propose to actually release 0.9 at > the > > > > > > beginning > > > > > > > > > of November. > > > > > > > > > This release is to verify our distribution with both the > > > Polaris > > > > > > > > > community and also the Apache Incubator (IPMC). > > > > > > > > > - After 0.9 release, I propose to move forward to 1.0 > release > > > (I > > > > > > think > > > > > > > > > it would be great to have 1.0 release before end of the > year) > > > > > > > > > - I also think we should use the "fast" major version as > soon > > > as we > > > > > > > > > introduce any breaking change. I don't see a problem with > > > having > > > > > > > > > Polaris 1, Polaris 2, Polaris 3, ... quickly if needed. > > > > > > > > > > > > > > > > > > Regards > > > > > > > > > JB > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
