Hi Jarek,
I'd also favor option 2...
one idea (into the blue?) would it work instead of making
`apache-airflow-providers-common-compat>=1.7.4 # TODO: bump`
to use the math expression and use this as marker like:
`apache-airflow-providers-common-compat>1.7.4` <-- greater not greater
or equal
Not sure whether this makes "new" problems in dev-env as breeze locally
might fail as >1.7.4 is just not existing or will it still use it from
repo/main? Just tesintg it with a small change on main generates an
error with: apache-airflow-providers-standard 1.9.0 has requirement
apache-airflow-providers-common-compat>1.7.4, but you have
apache-airflow-providers-common-compat 1.7.4.
Jens
On 21.10.25 08:47, Jarek Potiuk wrote:
Hello here,
We have been, over the last few months. struggling with an approach for
bumping versions of providers, when there are inter-dependencies between
providers. We have a very good example of it now when we want to move
Airflow 3 SDK compatibility code to common.compat provider (example for
standard provider herehttps://github.com/apache/airflow/pull/56867) and I
would like to use that opportunity to discuss possible approaches and
settle on a good solution that Elad and potentially other release manager
could follow.
Also that might be an input later to the "potential providers separation"
discussion we might have later this year as it describes one of the issues
that happens when we isolate things to separate repos (but this is
something we will deal with in the future).
# Problem
We have provider A depending on B and we want to add a code to provider B
that A will use. In this case, we add the "sdk" module to provider A.
Example: we add "compatibility" code to "common.compat" provider (currently
released version 1.7.4) and in the upcoming "standard" provider we want to
use that code. This means that when we release standard provider with that
code we have to have this dependency:
"apache-airflow-providers-common-compat >= FUTURE VERSION OF COMMON
COMPAT".
Due to the way Python dependency system works and development installation
of dependent providers work in development workspace, we cannot just say
(without actually bumping the common.compat in our code to 1.8.0), this
will cause conflict.
""apache-airflow-providers-common-compat >= 1.8.0"
# Possible solutions
We tried two different things:
1) having contributors bump "common.compat" when they add new things.
This is problematic, because bumping the version is a release
manager's work, and it gets in the way of preparing release notes. Release
notes are prepared at the time of release preparation and we have
automation that helps the release manager to decide what version we should
bump provider to - based on types of merged changes. Imagine that we have a
breaking change, then the new version would be a "2.0.0" not "1.8.0" - but
the release manager knows that only after all changes since last release
are merged. Also our automation properly prepares release notes in this
case, grouping the changes by type, and it is part of the "bumping version"
process - but once the version has been bumped, that is already not easy to
reconciiliate.
2) Do NOT bump the version by contributors - but leave a TODO to do so by
Release manager:
"apache-airflow-providers-common-compat>=1.7.4", # + TODO: bump to next
version
While this is a bit "hacky" - it kind of works and we can even easily
automate it during the release process. Say - we change if "TODO: bump" is
present as comment and we automatically bump it a release preparation time
(we would have to make topological sort and first bump dependent providers
and then those that depend on them (to know the version to bump to) but
this should be **easy"".
Also - we could make the same approach for breaking changes (contributors
currently are supposed to add migration comments at the top of release
notes when the next version of the provider is going to be breaking, the
contributor could then add a "# TODO: bump major version of provider as
there are breaking changes" comment.
3) Maybe there are other ways we have not thought about ?
# Question
I think - personally - that option 2) is better as it completely leaves the
"version bump" in the hands of the release manager - which seems more
robust and less brittle. But it has a small problem that contributors need
a bit of education and explanation on how to do it (it's not obvious). We
could add some comments to the pyproject.toml to explain it - but no-one
reads those comments anyway - so it will then be up to committers to learn
it as well and make it part of the review process to make sure this is
happening.
I would love to hear your thoughts about it - especially when we have
discussion to pick-up about making possible to accept more providers -
clarifying this part of the process and settling down on how we approach
this use case (which is actually quite common - not only in common.compat
and common.*, but also in some cases where for example Google provider
depends on some features in cncf.kubernetes etc.
WDYT?
J.
