API/model (r)evolution and functionalities are different topics, but
very related when it comes to (semantic) versioning and releases.
(**flame off**) IMHO versions are rather just arbitrary,
somewhat subjectively chosen numbers, sometimes driven by marketing-y
thoughts... (**flame on**)
Re: Maintaining major+minor version-branches: I'd also like to not have
to maintain more than one branch, because backporting changes can be a
pain. However, I also think that it would be wrong to not have the
_ability_ to maintain multiple version branches, even if it's just for
relevant security updates or critical bug fixes. Having (too) many
active release version branches also lets an
compatibility/upgrade/downgrade matrix explode.
I think that we should at least try to stick with a single branch and
only release from there. Or a mixed model, where there are version
branches, but those only get (patch) version dependency upgrades -
mostly automated.
Re: API compatibility: I think, it's worth to define what a user-facing
API is. For me, it's really only the user-facing and documented REST
API(s) and nothing more. Those user-facing Polaris APIs must be designed
with backwards/forwards compatibility in mind. Assuming that the (REST)
API resources and service interfaces and service implementations are
"properly" abstracted and there's good-enough test coverage, it's
possible to do a "major REST API version bump" and maintain those for
quite some time.
Re: Deprecating (for removal): this is probably a case-by-case decision.
Whole REST API endpoints need to be retained much longer (years maybe?).
Some administrative functionality however can potentially be
removed/replaced relatively quickly.
SPIs, think: interfaces that are used by Polaris extensions/plugins, for
example pluggable authN/Z, have a different compatibility guarantee -
rather a "best effort" approach based on technical
needs/requirements/efforts. All other code, that's not explicitly
declared to be some kind of SPI, is *not* a public API. The Java
'public' modifier does not make a Java type a public API. Unlike
Iceberg/Parquet/etc, Polaris code is "only" a server/service
implementation - but not a dependency that users pull into their projects.
Backend (database persistence) however is another different animal.
Upgrades and downgrades and concurrently running mixed-version scenarios
must work.
Robert
On 24.10.24 19:25, Yufei Gu wrote:
I believe it would be helpful to clearly define the following for our API
management along with versioning:
1. *Valid APIs*: It's common to support multiple versions of a valid
API. For example, both v1 and v2 of loadTable should remain functional.
2. *Deprecated APIs*: Even after deprecation, APIs should continue to be
supported to avoid disruptions.
3. *Removing Deprecated APIs*: We need a careful approach for retiring
deprecated APIs, as removing them can cause breaking changes for existing
clients. This step should be treated as a breaking change and handled with
caution.
Additionally, it is helpful to differentiate between *specs* (e.g., REST
endpoints) and *APIs* (public classes and methods). REST endpoints tend to
have longer lifecycles and will be deprecated or removed according to a
well-defined plan(discussion, announcement, etc), while public methods
should adhere to semantic versioning.
Yufei
On Wed, Oct 23, 2024 at 4:55 AM Jean-Baptiste Onofré <j...@nanthrax.net>
wrote:
By experience, it's very difficult to maintain two API versions in the
same codebase: very soon, we have a lot of "conditional statements" in
the code, which is not super elegant. Supporting previous API versions
(with routing) is possible as soon as we don't introduce "conditional
statements" per version.
The maintenance of branches is not necessarily painful: all new
features and improvements go on the latest active branch, other
branches just receive security fixes, dependency updates or major
fixes.
Else, we clearly state that we support only the latest active branch.
Several Apache projects maintain a couple of active branches (Apache
Camel, Apache ActiveMQ, etc).
I think it makes sense to bump the Polaris version with the Iceberg
REST Spec version. The versions don't necessarily match (like Polaris
3 can implement Iceberg REST Spec 2), but any new major REST Spec
version should result in a new major Polaris version.
Regards
JB
On Wed, Oct 23, 2024 at 1:05 PM Michael Collado <collado.m...@gmail.com>
wrote:
Personally, I’d rather not maintain two branches. I think it’s easier to
maintain two API versions in the same codebase rather than have to back
port changes into stale branches.
This also leads to the question of what we do for Iceberg spec API
version
changes. I can imagine if the whole REST spec changes to v2, we might
release Polaris 2.0, but what do we do if Iceberg only increments the
version for one or two APIs?
Mike
On Tue, Oct 22, 2024 at 5:42 AM Jean-Baptiste Onofré <j...@nanthrax.net>
wrote:
Good point.
Imho, we should increment API versions when we introduce "breaking
changes".
On major versions, I don't see a need to support previous API
versions. Users who want to support previous API can always use
previous Polaris versions.
For us, we can "maintain" two major versions in parallel (the last one
with active new features, improvements, and bug fixes on both the last
version and previous one).
For instance, in the Apache ActiveMQ project maintains both ActiveMQ
5.x and 6.x, with complete different API support (ActiveMQ 5.x
implements javax.jms 1.1, whereas ActiveMQ 6.x implements jakarta.jms
3.1).
So, regarding your example, I would say that Polaris 2.0 should not
necessary support v1 APIs, but focus on vX APIs and important changes.
We can still maintain Polaris 1.x for v1 API (with dependency updates
and bug fixes).
Regards
JB
On Tue, Oct 22, 2024 at 12:10 AM Michael Collado <
collado.m...@gmail.com>
wrote:
What's the support policy for API versions? Do we increment API
versions
for breaking API changes (this seems reasonable and likely)? And do
we
support previous API versions for some period of time?
If the answer to both of those questions is "yes" , then what is the
purpose of fast major verison releases? If we ship Polaris 2.0,
shouldn't
the service still support the v1 APIs? For me, a 2.0 release should
mean
pretty radical changes - new features, such as non-Iceberg table
support,
delegated Authz policies, etc.
On Mon, Oct 21, 2024 at 11:53 AM Yufei Gu <flyrain...@gmail.com>
wrote:
I'm not opposed to rapid evolution, but I’m trying to understand
the
reasoning behind the fast major version. It’s ultimately a tradeoff
between
maintaining backward compatibility and advancing quickly through
breaking
changes—we can’t achieve both at the same time. A faster major
version
release seems to indicate a preference for rapid progress over
stability.
Otherwise, why push for it? Am I missing something?
Yufei
On Mon, Oct 21, 2024 at 11:26 AM Dmitri Bourlatchkov
<dmitri.bourlatch...@dremio.com.invalid> wrote:
IMHO, production compatibility concerns are valid, but they are
not
related
to the versioning scheme.
Backward-compatibility breakage should be declared anyway,
whether
the
move
is from 0.42.1 to 0.43.0 or from 2.1.1 to 3.0.0.
Still, bumping a major version indicates major change. It does
not
automatically indicate incompatibility. My reading of Semver [1]
guidelines
is that major must be incremented when incompatible changes are
present,
but there is no restriction on incrementing major and keeping the
product
backward compatible.
On the other hand, if we avoid major code changes because of the
risk of
incompatibility, that will slow down the project's evolution.
Cheers,
Dmitri.
[1] https://semver.org/
On Mon, Oct 21, 2024 at 2:07 PM Yufei Gu <flyrain...@gmail.com>
wrote:
+1 on moving forward with 0.9 as the first version!
I have some concerns about fast major versioning, especially
with
compatibility. Since the catalog is crucial in production,
teams
may be
reluctant to upgrade, particularly with breaking changes.
Does fast versioning help in this case, or could it make things
harder?
Does it encourage more frequent upgrades, and is there a plan
to
manage
production impact? While we can't completely avoid user forks,
we
want
to
minimize divergence.
Yufei
On Fri, Oct 18, 2024 at 6:33 AM Dmitri Bourlatchkov
<dmitri.bourlatch...@dremio.com.invalid> wrote:
+1 to "fast" major version.
Cheers,
Dmitri.
On Fri, Oct 18, 2024 at 5:17 AM Jean-Baptiste Onofré <
j...@nanthrax.net>
wrote:
Hi folks,
First of all, thanks everyone for the first community
meeting
yesterday. It was a great meeting, very interesting
discussions.
You can find the meeting notes on the Polaris website
(https://polaris.apache.org/community/meetings/):
https://docs.google.com/document/d/1TAAMjCtk4KuWSwfxpCBhhK9vM1k_3n7YE4L28slclXU/edit?usp=sharing
As discussed during the meeting, here's the release
proposal:
- the first Polaris release will be 0.9. I'm doing a new
pass
on
the
preparation steps (due diligence on
NOTICE/LICENSE/DISCLAIMER/dependencies, automatic
"scripts",
etc).
As
we are mid October, I propose to actually release 0.9 at
the
beginning
of November.
This release is to verify our distribution with both the
Polaris
community and also the Apache Incubator (IPMC).
- After 0.9 release, I propose to move forward to 1.0
release
(I
think
it would be great to have 1.0 release before end of the
year)
- I also think we should use the "fast" major version as
soon
as we
introduce any breaking change. I don't see a problem with
having
Polaris 1, Polaris 2, Polaris 3, ... quickly if needed.
Regards
JB
--
Robert Stupp
@snazy
