Thanks, Sumit, for bringing this up. re: *versioning for Python client*:
I favour a bit of a hybrid of 1 and 2. Initially, I favored just (1) when we talked but after seeing how Elastic-Search does it I liked their versioning strategy. How their python-client is compatible with ES itself is documented at https://elasticsearch-py.readthedocs.io/en/v7.12.0/#compatibility. Since Airflow from 2.0 strictly follows SemVer we can continue with the assumption that we (Airflow) won't break backwards-compatibility (to the best of our ability). So we can have *M.x.y *as our versioning for our client where *M *is the Major Version of the client *that matches the Major version of Airflow*; *x* is the minor version for the client & *y* is the patch release. Here *x* & *y* are independent of Airflow version. Our goal and message to user would be that "You have to use a version of client library matching the major version of Airflow" re: *the release of different client libraries*: Client for each language should be versioned & released separately. Regards, Kaxil On Wed, Mar 31, 2021 at 7:01 PM Jarek Potiuk <ja...@potiuk.com> wrote: > I think there are more questions we need to answer when it comes to > different versions. If we are going to different versioning for agent and > Airflow, we will have to accept the fact that over time people will have a > "matrix of versions" - for both Airflow and the client. And this introduces > "dependency mess" if we do not handle it well. > > I think in this we should give the user much clearer indication on which > version of Airflow the client should work on, and (more important) which > version it does NOT work on and which features will NOT be available. Both > cases are possible. > > There should be some way of knowing for the user, (in case there is a > backwards incompatibility or in case some features are not available) that > a given version of the client does not work with the version of Airflow > that the user has. > > I think users looking at the documentation should be able to be able to > figure out: > > 1) Will my client be compatible with the version of Airflow ? > 2) Will feature (API endpoint) x available in this version of client be > working with the version of Airflow I have? > 3) Which version of Airflow should I install in order to use feature X > that I have in this new version of client ? > > Similarly proper error messages should be displayed if there is > incompatibility or when a feature X is not available when you try to use a > version of client that supports it with Airflow version of Agent that does > not have it. > > So I'd say - separate versioning is fine as long as we handle the above. > > > Saying all that - there is a beautiful "maintenance" simplicity if we > actually synchronize the version and we say "Always use SAME version of > client as Airflow" (and we fail the client if the versions is different). > This is way more simple in communicating to the users, and with most > deployments, I think upgrading airflow and clients at the same time should > be no problem. > > I think over time as the matrix of potential Airflow/Client combinations > will grow this might mean bigger number of issues that we have to handle > ("Why ma client does not do this and this (and of course it will not have > neither client nor airflow version specified and we will have to ask for > clarification). > > So as I see it - maybe release-wise it's a little redundant and you'd have > to release a bugfix version of Airflow if you find a critical problem in > Client (which should be rare as agent is mostly automatically generated), > but synchronizing versions is very tempting from the "supportability" point > of view. > > Those are my 2 cents, I was not involved too much in the client part, so I > do not feel like I have a strong voice in there, but if I would vote > between the two - I'd choose synchronizing the releases. > > BTW. In providers we have a very different story - we keep dependency to > airflow via PIP (which we can set it, if we find a provider stops being > compatible with a version of Airflow). Also we automatically test in CI if > the provider installs and imports on Airflow 2.0.0 to verify that it is > backwards compatible (at least to the level of importability). This has > already saved us from "accidental" incompatibilities (wit yaml package > import change recently). So there, separate versioning makes much more > sense - also because the version of the provider relates exclusively to the > feature that the provider version has, not Airflow itself). I think this is > quite a different story from the client, which is very closely coupled with > Airflow. > > J. > > > > On Wed, Mar 31, 2021 at 11:08 AM Deng Xiaodong <xd.den...@gmail.com> > wrote: > >> Yep, if we can, I would prefer to explicitly mention something like 2.0.X >> :) >> >> On Wed, Mar 31, 2021 at 10:54 AM Sumit Maheshwari <sumeet.ma...@gmail.com> >> wrote: >> >>> Yeah totally agrees with you XD about the range vs the fixed version of >>> the main software. And with Airflow 2 dot & beyond, we would be following >>> the semantic versioning more judiciously, so with that can we assume that >>> users would be cognizant of the related Airflow version implicitly, or do >>> you think we should mention something like *2.0.X* in the changelog >>> explicitly? >>> >>> On Wed, Mar 31, 2021 at 1:38 PM Deng Xiaodong <xd.den...@gmail.com> >>> wrote: >>> >>>> Thanks, Sumit, for bringing this up. >>>> >>>> - Regarding *versioning for Python client*: I favour the 1st way as >>>> well. The only minor comment I have is: instead of "*mention the >>>> release version of the main software*", possibly what users would >>>> expect is a "*range of compatible versions of main software*". If we >>>> follow semver for Airflow itself relatively strictly, this should not be >>>> hard. >>>> >>>> - Regarding *the release of different client libraries*: I agree with >>>> you. They should be versioned & released separately. >>>> >>>> >>>> Regards, >>>> XD >>>> >>>> On Wed, Mar 31, 2021 at 9:21 AM Sumit Maheshwari < >>>> sumeet.ma...@gmail.com> wrote: >>>> >>>>> Thanks QP for your feedback. I think the latest release of K8s client >>>>> (v17.x.y) is moving in another direction. >>>>> >>>>> On Wed, Mar 31, 2021 at 12:13 AM QP Hou <q...@scribd.com.invalid> >>>>> wrote: >>>>> >>>>>> Thanks Sumit for kicking off this discussion! >>>>>> >>>>>> I am in favor of approach #1 as well. Airflow software version can >>>>>> change due to none API related changes, but the client should only >>>>>> care about the public API contract. I believe this is also the same >>>>>> approach that the k8s python client took as well? For example, the k8s >>>>>> python client version 12.0.1 targets k8s api version 1.16.15. >>>>>> >>>>>> I am also with you that clients for different languages should have >>>>>> their own release cycles. Other than testing burdens, every client >>>>>> could introduce their own bug fixes and breaking changes without >>>>>> having to trigger releases for all other clients. It would be a waste >>>>>> of work for us to bump the python client version just for a bug fix in >>>>>> the go client. >>>>>> >>>>>> Thanks, >>>>>> QP >>>>>> >>>>>> On Tue, Mar 30, 2021 at 1:56 AM Sumit Maheshwari <msu...@apache.org> >>>>>> wrote: >>>>>> > >>>>>> > Hello Airflow dev community >>>>>> > >>>>>> > I'm writing this mail in regards to seek feedback on the first >>>>>> release of Apache Airflow's Python client. This is an icebreaker email >>>>>> only >>>>>> and we will follow the Apache process of releasing the version later on. >>>>>> As >>>>>> of now, I've raised a small PR to add the missing files in the repo, >>>>>> which >>>>>> are required to do the release of the client lib. >>>>>> > >>>>>> > The main question I've in mind about the versioning. I see that >>>>>> there are mainly two theories related to maintain the release versions of >>>>>> clients: >>>>>> > >>>>>> > The first is about following the semantic release of the clients on >>>>>> its own. The release version of the client doesn't include the version of >>>>>> the software itself. Something like we are doing with Airflow's provider >>>>>> packages. However, in each release, we mention the release version of the >>>>>> main software (Airflow in this case). This is the simple most way of >>>>>> maintaining releases and followed majorly. >>>>>> > The second way is to include the software's version as well in its >>>>>> client's versioning, something like it's been done in the K8s Python >>>>>> client. In this approach, the client version provides more implicit >>>>>> knowledge about the compatible software, however it creates an extra >>>>>> burden >>>>>> of releasing the client with every minor release of the software, >>>>>> otherwise >>>>>> the versioning would fall apart. Also, there is a good chance that most >>>>>> of >>>>>> these minor or even a major release has no changes in the APIs, but the >>>>>> client would be needed to re-released nevertheless, just to keep the >>>>>> versioning in check. >>>>>> > >>>>>> > Personally, I'm in favour of the 1st way, as it's simple to manage >>>>>> and doesn't tightly couple client with its software. >>>>>> > >>>>>> > Another unrelated question I have is about the release of different >>>>>> client libraries. As of now, Airflow has Python and Go clients, so should >>>>>> we try to release them together or they should follow their own paths? In >>>>>> general, I'm against batching them together, cause the person who is >>>>>> releasing them may not be able to test all the different clients in the >>>>>> future. >>>>>> > >>>>>> > Let me know your thoughts on these 2 points. >>>>>> > >>>>>> > Thanks, >>>>>> > Sumit Maheshwari >>>>>> > PMC Apache Airflow >>>>>> >>>>> > > -- > +48 660 796 129 >
