Hello everyone,
TL;DR; I think there is one thing about dependencies that we should have
some agreement on to make some common approach. Namely about using upper
bounds on our dependencies.
Should we set some rules on when we set upper-bounds on the deps ? What
rules should they be?
Currently we use constraints to make sure Airflow in specific version can
be installed using "golden" set of dependencies. Those are part of our CI,
automatically updated and tests which makes it really nice as they are
"fixed" for installation of particular version but they continuously
upgrade (even across major versions) when they pass tests in CI.
This all happens pretty much automatically by our CI, except the cases
where our dependencies are upper-limited. We occasionally do some
"setup.py", "setup.cfg" changes manually to bump some upper limits, but
this is more the result of some external request or "occasional" cleanup to
do it - rather than a regular process.
Now, we have some dependencies that are upper-bound for good reasons and
they are documented. We also have a number of dependencies that are not
upper-limited. But I think this pretty inconsistent. Small excerpt from
setup.cfg:
# Logging is broken with itsdangerous > 2
itsdangerous>=1.1.0, <2.0
# Jinja2 3.1 will remove the 'autoescape' and 'with' extensions, which
would
# break Flask 1.x, so we limit this for future compatibility. Remove
this
# when bumping Flask to >=2.
jinja2>=2.10.1,<3.1
...
# python daemon crashes with 'socket operation on non-socket' for
python 3.8+ in version < 2.2.4
# https://pagure.io/python-daemon/issue/34
python-daemon>=2.2.4
python-dateutil>=2.3, <3
python-nvd3~=0.15.0
* itsdangerous is upper limited and the reason is specified in the comment
(though we do not know when we could remove the limit)
* Jinja is upper-limited and we know not only why but also when it can be
removed
* Python-daemon is not upper-limited but it has a comment why it is
"lower-limited" (which is pretty useless IMHO)
* python-dateutil is upper-limited but we do not know why
* python-nvd3 is also upper limited (~0.15.0 - limits it to any 0.15.x
version but 0.16 could not be installed)
I think there are many inconsistencies and the way we treat dependency
limits is pretty inconsistent - we have no rules agreed.
I would love to discuss how we can standardize it or at least set some
rules we can follow.
My initial thoughts are:
* we can (but do not have to) have lower bounds if we need to protect and
we know about some limitations, but we do not need to document those. Just
set the limit. The nice thing about lower bounds that they do not "decay"
over time. By default latest "eligible" version is installed by PIP (but of
course if other deps have conflicting upper bounds, keeping lower limits
when unnecessary is a problem.
* by default we should not have upper-bounds. We know it limits our users
and constraints + CI tests are nicely handling the scenario when things are
breaking.
* we should only introduce upper bounds if we know that there is breaking
change (or that it is very likely and "foreseen" - betas/rc2/discussions
about upcoming breaking changes in the dependencies
* when we introduce upper-bounds we should always specify why and what is
the condition to remove them
WDYT?
J.
