I think there are more questions we need to answer when it comes to
different versions. If we are going to different versioning for agent and
Airflow, we will have to accept the fact that over time people will have a
"matrix of versions" - for both Airflow and the client. And this introduces
"dependency mess" if we do not handle it well.
I think in this we should give the user much clearer indication on which
version of Airflow the client should work on, and (more important) which
version it does NOT work on and which features will NOT be available. Both
cases are possible.
There should be some way of knowing for the user, (in case there is a
backwards incompatibility or in case some features are not available) that
a given version of the client does not work with the version of Airflow
that the user has.
I think users looking at the documentation should be able to be able to
figure out:
1) Will my client be compatible with the version of Airflow ?
2) Will feature (API endpoint) x available in this version of client be
working with the version of Airflow I have?
3) Which version of Airflow should I install in order to use feature X that
I have in this new version of client ?
Similarly proper error messages should be displayed if there is
incompatibility or when a feature X is not available when you try to use a
version of client that supports it with Airflow version of Agent that does
not have it.
So I'd say - separate versioning is fine as long as we handle the above.
Saying all that - there is a beautiful "maintenance" simplicity if we
actually synchronize the version and we say "Always use SAME version of
client as Airflow" (and we fail the client if the versions is different).
This is way more simple in communicating to the users, and with most
deployments, I think upgrading airflow and clients at the same time should
be no problem.
I think over time as the matrix of potential Airflow/Client combinations
will grow this might mean bigger number of issues that we have to handle
("Why ma client does not do this and this (and of course it will not have
neither client nor airflow version specified and we will have to ask for
clarification).
So as I see it - maybe release-wise it's a little redundant and you'd have
to release a bugfix version of Airflow if you find a critical problem in
Client (which should be rare as agent is mostly automatically generated),
but synchronizing versions is very tempting from the "supportability" point
of view.
Those are my 2 cents, I was not involved too much in the client part, so I
do not feel like I have a strong voice in there, but if I would vote
between the two - I'd choose synchronizing the releases.
BTW. In providers we have a very different story - we keep dependency to
airflow via PIP (which we can set it, if we find a provider stops being
compatible with a version of Airflow). Also we automatically test in CI if
the provider installs and imports on Airflow 2.0.0 to verify that it is
backwards compatible (at least to the level of importability). This has
already saved us from "accidental" incompatibilities (wit yaml package
import change recently). So there, separate versioning makes much more
sense - also because the version of the provider relates exclusively to the
feature that the provider version has, not Airflow itself). I think this is
quite a different story from the client, which is very closely coupled with
Airflow.
J.
On Wed, Mar 31, 2021 at 11:08 AM Deng Xiaodong <xd.den...@gmail.com> wrote:
> Yep, if we can, I would prefer to explicitly mention something like 2.0.X
> :)
>
> On Wed, Mar 31, 2021 at 10:54 AM Sumit Maheshwari <sumeet.ma...@gmail.com>
> wrote:
>
>> Yeah totally agrees with you XD about the range vs the fixed version of
>> the main software. And with Airflow 2 dot & beyond, we would be following
>> the semantic versioning more judiciously, so with that can we assume that
>> users would be cognizant of the related Airflow version implicitly, or do
>> you think we should mention something like *2.0.X* in the changelog
>> explicitly?
>>
>> On Wed, Mar 31, 2021 at 1:38 PM Deng Xiaodong <xd.den...@gmail.com>
>> wrote:
>>
>>> Thanks, Sumit, for bringing this up.
>>>
>>> - Regarding *versioning for Python client*: I favour the 1st way as
>>> well. The only minor comment I have is: instead of "*mention the
>>> release version of the main software*", possibly what users would
>>> expect is a "*range of compatible versions of main software*". If we
>>> follow semver for Airflow itself relatively strictly, this should not be
>>> hard.
>>>
>>> - Regarding *the release of different client libraries*: I agree with
>>> you. They should be versioned & released separately.
>>>
>>>
>>> Regards,
>>> XD
>>>
>>> On Wed, Mar 31, 2021 at 9:21 AM Sumit Maheshwari <sumeet.ma...@gmail.com>
>>> wrote:
>>>
>>>> Thanks QP for your feedback. I think the latest release of K8s client
>>>> (v17.x.y) is moving in another direction.
>>>>
>>>> On Wed, Mar 31, 2021 at 12:13 AM QP Hou <q...@scribd.com.invalid> wrote:
>>>>
>>>>> Thanks Sumit for kicking off this discussion!
>>>>>
>>>>> I am in favor of approach #1 as well. Airflow software version can
>>>>> change due to none API related changes, but the client should only
>>>>> care about the public API contract. I believe this is also the same
>>>>> approach that the k8s python client took as well? For example, the k8s
>>>>> python client version 12.0.1 targets k8s api version 1.16.15.
>>>>>
>>>>> I am also with you that clients for different languages should have
>>>>> their own release cycles. Other than testing burdens, every client
>>>>> could introduce their own bug fixes and breaking changes without
>>>>> having to trigger releases for all other clients. It would be a waste
>>>>> of work for us to bump the python client version just for a bug fix in
>>>>> the go client.
>>>>>
>>>>> Thanks,
>>>>> QP
>>>>>
>>>>> On Tue, Mar 30, 2021 at 1:56 AM Sumit Maheshwari <msu...@apache.org>
>>>>> wrote:
>>>>> >
>>>>> > Hello Airflow dev community
>>>>> >
>>>>> > I'm writing this mail in regards to seek feedback on the first
>>>>> release of Apache Airflow's Python client. This is an icebreaker email
>>>>> only
>>>>> and we will follow the Apache process of releasing the version later on.
>>>>> As
>>>>> of now, I've raised a small PR to add the missing files in the repo, which
>>>>> are required to do the release of the client lib.
>>>>> >
>>>>> > The main question I've in mind about the versioning. I see that
>>>>> there are mainly two theories related to maintain the release versions of
>>>>> clients:
>>>>> >
>>>>> > The first is about following the semantic release of the clients on
>>>>> its own. The release version of the client doesn't include the version of
>>>>> the software itself. Something like we are doing with Airflow's provider
>>>>> packages. However, in each release, we mention the release version of the
>>>>> main software (Airflow in this case). This is the simple most way of
>>>>> maintaining releases and followed majorly.
>>>>> > The second way is to include the software's version as well in its
>>>>> client's versioning, something like it's been done in the K8s Python
>>>>> client. In this approach, the client version provides more implicit
>>>>> knowledge about the compatible software, however it creates an extra
>>>>> burden
>>>>> of releasing the client with every minor release of the software,
>>>>> otherwise
>>>>> the versioning would fall apart. Also, there is a good chance that most of
>>>>> these minor or even a major release has no changes in the APIs, but the
>>>>> client would be needed to re-released nevertheless, just to keep the
>>>>> versioning in check.
>>>>> >
>>>>> > Personally, I'm in favour of the 1st way, as it's simple to manage
>>>>> and doesn't tightly couple client with its software.
>>>>> >
>>>>> > Another unrelated question I have is about the release of different
>>>>> client libraries. As of now, Airflow has Python and Go clients, so should
>>>>> we try to release them together or they should follow their own paths? In
>>>>> general, I'm against batching them together, cause the person who is
>>>>> releasing them may not be able to test all the different clients in the
>>>>> future.
>>>>> >
>>>>> > Let me know your thoughts on these 2 points.
>>>>> >
>>>>> > Thanks,
>>>>> > Sumit Maheshwari
>>>>> > PMC Apache Airflow
>>>>>
>>>>
--
+48 660 796 129
