The new docker provider will still install and work on Airflow 2.1 --
the only change is to add a new field to the provider info dict
(`task-decorators`) that Airflow 2.2 will read, but that Airflow 2.1
will use.
If you try to use `@task.docker` in Airflow it will fail with
"_TaskDecorator has no property docker" which I think is fair -- and
the docs about this feature say
(<http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow/latest/tutorial_taskflow_api.html#using-the-taskflow-api-with-docker-or-virtual-environments>):
> If you have tasks that require complex or conflicting requirements
then you will have the ability to use the Taskflow API with either a
Docker container (since version 2.2.0) or Python virtual environment
(since 2.0.2).
So I say we should not change the minimum version of the provider and
just bump the minor version as normal when releasing a new feature --
yes, that new feature isn't available on Airflow 2.1.x but nothing
would break so I think this is fine.
-ash
On Thu, Sep 30 2021 at 09:56:23 +0200, Kamil Breguła
<[email protected]> wrote:
If we release this package, the user will not be able to test it in
any released version as such a version does not exist yet. This could
also potentially mean that it will never work, as unreleased versions
of Airflow may have breaking API changes. We only maintain API
compatibility between released versions of Airflow. Airflow developer
versions may have breaking changes, and that's fine as long as this
issue is fixed before a new version is released.
czw., 30 wrz 2021 o 08:32 Kaxil Naik <[email protected]
<mailto:[email protected]>> napisał(a):
Agree with Kamil here. We should just adjust the versioning of the
docker provider and bump the Minor version to indicate New Feature.
To XD's question - we can release a patch version in that minor
version series for Docker Provider if we find a critical fix.
Regards,
Kaxil
On Thu, Sep 30, 2021, 04:03 Jarek Potiuk <[email protected]
<mailto:[email protected]>> wrote:
I have no strong opinion on this, but would like to have
clarification on one question: I understand you highlighted there
is no changes nor fixes in the provider, but what if in the future
we need to fix something for the “classic” operator which may
still be used in Airflow 2.1 or even lower version installations?
How will releases etc. be managed in such cases?
Yep. Good question. This is about the only "potential" issue -
currently we have no process for that as we've never had to do it.
However it is always possible to do it if and when we need it. If
we find that there is a critical fix for Docker Provider that we
would need to fix, we can always branch-off from the last released
version that works for 2.1 and cherry-pick and release a patched
version.
We did not see a need for that so far. We released all providers
when we released 2.1.0 which are 2.1+ only (because of
apply_defaults) and so far there was not a case where anyone raised
a need to release any bugfix for previous versions. With our SemVer
approach anyone should be "SAFE" migrating to newer version of
Airflow (even 2.2) as they are all supposed to be backwards
compatible, so I think our approach should be "If you need anything
that we release for the DockerProvider in the future - migrate to
2.2. But this is a "default" approach I think and I see the case
of a really critical issue that will result in releasing a
patchlevel version for the previous release.
Again - we have not done that yet, and we want to avoid it for as
long as we can, but there is nothing (except the overhead on
cherry-picking and releasing it) to prevent us from doing so. We
have all history, tags, changelogs and mechanisms to selectively
release only one provider if needed. But I think it should be
really Ad-Hoc and when we REALLY REALLY need it.
J.
I may have misunderstood or missed something, and be asking a
dumb question. Please correct me in that case.
Thanks!
XD
On Wed, Sep 29, 2021 at 22:15 Jarek Potiuk <[email protected]
<mailto:[email protected]>> wrote:
Hello Everyone,
I am just about to release the September Providers, but Kamil
raised a valid concern and he convinced me that it might be a
problem if we release a new Docker Provider with minimum airflow
version 2.1.
The new provider has the new "Docker Decorator" feature which
will be available only in 2.2. But other than that - it is
backwards compatible - so you could install it on Airflow 2.1 and
have the "classic" operator work. But if you try to use the
decorator, you will be able to write a Dag with this decorator,
but it simply won't work (and you will have no message about it).
The Docker Provider has no other changes nor fixes.
I am tempted (and quite convinced) to increase the min airflow
version of the Docker Provider to 2.2 so that it can be only
installed there (and we make it ready for the 2.2 release).
Does anyone have a strong opinion on that ?
J.
