> Exactly to remove the dev requirements airflow has from interfering with
our own dev requirements. Though this approach notably doesn't change any
of the used non-dev requirements, it's not clear to me (and I'm probably
just missing something) why your diff includes non-dev requirements.
The thing is that I do not "remove" devel features - because due to
transitive dependencies I would have to go deep down the rabbit hole which
dependencies are "devel" and which not. So what I do, I ask `uv` to do the
job for me and repeat the resolution but with the `--exact` flag (combined
with the already used `--resolution higest` and `--no-sources`).
Apparently in this case `uv` will sometimes decide that it's ok to bump
some dependencies that are left after initial resolution. The thing is that
uv will not install some versions of some packages if they are conflicting
with already installed packages. What I believe `uv` does when I use
the *--exact
*flag of `uv pip install` it "fixes" the initial resolution. It finds out
what is needed, removes those packages that are not needed, and then it
resolves again and improves the resolution (that would be my guess at least
- it could be done slightly differently and in a single step I guess.
Essentially the difference between the left and right side is:
a) install absolutely everything from the workspace
b) do this:
left: uv pip install --no-sources 'apache-airflow[all]'
'apache-airflow-core[all]' apache-airflow-task-sdk ./airflow-ctl
--reinstall --resolution highest --find-links file:///dist
right: uv pip install --no-sources *--exact --strict* 'apache-airflow[all]'
'apache-airflow-core[all]' apache-airflow-task-sdk ./airflow-ctl
--reinstall --resolution highest --find-links file:///dist
In the current state content of the fille://dist should be the same in both
cases:
apache-airflow-providers-edge3 (1.1.2): wheel -
apache_airflow_providers_edge3-1.1.2-py3-none-any.whl
apache-airflow-providers-keycloak (1.0.0): wheel -
apache_airflow_providers_keycloak-1.0.0-py3-none-any.whl
apache-airflow-providers-snowflake (6.5.1): wheel -
apache_airflow_providers_snowflake-6.5.1-py3-none-any.whl
This is a bit of in-process quirk of the "PyPI" resolution - those are the
three packages that we have updated versions of in the sources, but they
are not released yet. But they should be essentially the same in both
cases. So the difference between left and right is `--exact --strict`
(strict I added to make sure that we are not removing something that is
still needed).
J.
On Tue, Jul 22, 2025 at 5:22 PM Damian Shaw <ds...@striketechnologies.com>
wrote:
> As I user of Airflow I have ended up implementing this for my environments:
>
> pip install apache-airflow[all or select-extras] -c {official constraints}
> > smaller-run-time-constraints.txt
> pip-compile {all my requirements including development}.txt -c
> smaller-run-time-constraints.txt
>
> Exactly to remove the dev requirements airflow has from interfering with
> our own dev requirements. Though this approach notably doesn't change any
> of the used non-dev requirements, it's not clear to me (and I'm probably
> just missing something) why your diff includes non-dev requirements.
>
> Damian
>
> -----Original Message-----
> From: Jarek Potiuk <ja...@potiuk.com>
> Sent: Tuesday, July 22, 2025 11:04 AM
> To: dev@airflow.apache.org
> Subject: [DISCUSS] Removing dev dependencies from PyPI constraints ?
>
> Hello here,
>
>
> *Tl;DR; We had an interesting discussion on Slack today and I wanted to
> bring it here to discuss what we should do - whether we should keep devel
> dependencies in our PyPI constraints or not.*
>
> *A bit of context:*
> In our CI we generate several types of constraints. one of them are
> "source" constraints that we use during our CI testing - including all our
> testing tools, mypy, ruff etc., and another type is "PyPI" constraint -
> which is more <What should be the "golden" versions of dependencies when
> you install Airflow using `pip install` for production>.
>
> The main difference between the two (roughly) is that "source constraints"
> only uses dependencies from "branch tip" ("main", "v3-0-test"), where
> "PyPI constraints" uses "Airflow core" branch-tip dependencies - but all
> the providers are installed from PyPI. There are few nuances when some
> packages are being released but basically - for example for Airflow 3.0.3,
> "PyPI"
> constraints is "what is the current set of dependencies and providers when
> we install airflow from PyPI at the moment of release".
>
> So target for "source" constraints are "Airflow contributors" where target
> for "PyPI constraints" are "Airflow users".
>
>
> *The problem*
> In the slack discussion
> https://apache-airflow.slack.com/archives/C06K9Q5G2UA/p1753169914722049 -
> slack user @dashmug (sorry - do not know name) noticed that the "PyPI"
> constraints also contain Airflow development dependencies listed (like
> ruff or mypy). That made me think a bit and I quickly came up with PR that
> could "fix" it - i.e. only keep the "real" constraints in.
> https://github.com/apache/airflow/pull/53631 - it's as easy as adding the
> `--exact` (and `--strict` for verification) flag to the `uv pip install`
> command we use.
>
> I agree it's a bit confusing to have the devel dependencies there - it is
> not the "intention", but there is very little problem with it - the fact
> that mypy or ruff are listed with a version in constraints, does not mean
> it must be installed - constraints mainly say "if you install this, it
> should be this version" . And there is no problem whatsoever (and it is
> even recommended by us) to have "airflow with its deps" installed with
> constraints, and then anything else you need to install or update -
> without. So other than potential confusion, it usually does not cause harm.
>
> Now... It might seem that removing devel deps has no side effects and we
> should **just do it** - but it's not that easy. Main problem is that those
> devel dependencies might "hold" other dependencies (directly or
> transitively) from being upgraded - and when we remove the devel
> dependencies and do the resolution again - we might have a DIFFERENT set of
> dependencies, simply because removing the devel dependencies, might "free"
> the other deps and they might get upgraded. Yeah - I know it's not obvious
> for someone who did not spend 3 years solving dependency issues :).
>
> And the problem with it is that because we need those devel deps to test
> things - the upgraded dependencies might have never been tested in our CI -
> because we never upgraded to later versions. It's a bit of
> Heisenberg-effect. By using test tools we are changing the state of the
> tested thing. Currently our "PyPI" constraints reflect the state of Airflow
> "when the testing tools are installed" - and we are generally not able to
> test the state of Airflow "without the testing tools" - by the sheer fact
> that we do not have those tools :)
>
> Just to illustrate the case - you can see what happens when we removed
> devel deps:
> https://github.com/apache/airflow/actions/runs/16444553234 - in summary
> you can see a colored version that is a bit clearer but I dumped the diff
> below from an example - Python 3.10 case. Left side < is "what was with
> testing tools" and > is without them. You can see a LOT of test tools
> removed (pytest, ruff and others - including dependencies used only by
> those tools)
>
> But also you can see (likely some of the devel/test deps limited those)
>
> * Authlib - upgraded from 1.3.1 to 1.6.1
> * Google - genai upgraded from 1.2.0 to 1.20.0
> * Httpx - upgraded from 0.27.0 to 0.28.1
> * Pinotdb - upgraded from 5.6.0 to 5.7.0
> * Sentry-sdk - upgraded from 2.33.1 to 2.33.2 (this is false positive as
> it has just been released and new version was picked)
> * validators - upgraded form 0.34.0 to 0.35.0
> * weaviate-client - upgraded from 4.9.0 to 4.16.3
> * websockets - upgraded from 14.2 to 15.0.1
>
>
> This means that all the "upgraded" packages above have not been - likely
> ever tested in our CI.
>
> And when we put them in "versioned" constraints, it might mean that they
> will be obviously failing.
>
> So ... there is a risk connected with just generating the "non-devel"
> constraints without additional tests. But that also means that the user
> who does not use constraints might get those versions anyway - default
> behaviour of `pip install airflow` will be to install those upgraded
> versions above if you do not use constraints. But also that is no different
> than installing a released airflow version a few weeks or months after
> release without constraints - there will be tens of packages that were
> released since our release and installing airflow without constraints will
> pick those.
>
> However - when we manually install RC candidates - say by running airflow
> from rc1 image, we will use those "upgraded" dependencies - so some
> "obvious" problems might be caught during the rc manual testing - where you
> do not use test tools but run airflow "as if" it was installed in
> production.
>
> *The solution ? *
>
> There are few ways we can proceed with it:
>
> a) leave it as is - no big harm, potential confusion, by people who see it
> - potential differences
>
> b) remove devel following my PR and take the risk that we never tested (in
> CI) the dependencies we set in the constraints. That's a bit risky, but we
> can always reactively downgrade some of those packages after the fact if
> users will start reporting the issues
>
> c) potentially most complex thing - trying to remove devel dependencies
> somewhat combining the two - for example removing all the deps that would
> be removed in b) from a) - but that has also some edge cases - when
> different versions of dependencies can add or upgrade their dependencies,
> we are very likely to occasionally remove or add too much.
>
>
> I wonder (for those who managed to read that far - likely not many - what
> do you think here? What would be the best option ? Maybe there are other
> options I have not thought about ?
>
>
> J
>
>
> -----
>
> The diff for Python 3.10
>
>
>
> 34c34
> < Authlib==1.3.1
> ---
> > Authlib==1.6.1
> 64d63
> < Sphinx==8.2.3
> 78d76
> < aioresponses==0.7.8
> 82d79
> < alabaster==1.0.0
> 198d194
> < arro3-core==0.5.1
> 202d197
> < astroid==3.3.11
> 211,212d205
> < aws-sam-translator==1.99.0
> < aws-xray-sdk==2.14.0
> 258,259d250
> < cfgv==3.4.0
> < cfn-lint==1.38.0
> 262d252
> < checksumdir==1.2.0
> 277d266
> < coverage==7.9.2
> 291d279
> < deltalake==1.1.0
> 293d280
> < diagrams==0.24.4
> 301,302d287
> < docutils==0.21.2
> < duckdb==1.3.2
> 308d292
> < eralchemy2==1.4.1
> 311d294
> < execnet==2.1.1
> 321,322d303
> < flit==3.12.0
> < flit_core==3.12.0
> 386c367
> < google-genai==1.2.0
> ---
> > google-genai==1.20.0
> 389d369
> < graphql-core==3.2.6
> 398d377
> < grpcio-tools==1.62.3
> 403,404d381
> < hatch==1.14.1
> < hatchling==1.27.0
> 413c390
> < httpx==0.27.0
> ---
> > httpx==0.28.1
> 418d394
> < hyperlink==21.0.0
> 421,423d396
> < icdiff==2.0.7
> < id==1.5.0
> < identify==2.6.12
> 426d398
> < imagesize==1.4.1
> 430d401
> < incremental==24.7.2
> 433,435d403
> < iniconfig==2.1.0
> < inputimeout==1.0.4
> < ipdb==0.13.13
> 449d416
> < joserfc==1.2.2
> 451d417
> < jsonpatch==1.33
> 455,456d420
> < jsonpointer==3.0.0
> < jsonschema-path==0.3.4
> 463d426
> < kerberos==1.3.1
> 465,466d427
> < keyrings.alt==5.0.2
> < kgb==7.2
> 494,495d454
> < mmh3==5.1.0
> < mongomock==4.3.0
> 497,498d455
> < moto==5.1.8
> < mpmath==1.3.0
> 508,512d464
> < mypy-boto3-appflow==1.39.0
> < mypy-boto3-rds==1.39.1
> < mypy-boto3-redshift-data==1.39.0
> < mypy-boto3-s3==1.39.5
> < mypy==1.17.0
> 514c466
> < mysql-connector-python==9.3.0
> ---
> > mysql-connector-python==9.4.0
> 521,523d472
> < networkx==3.5
> < nh3==0.3.0
> < nodeenv==1.9.1
> 528,529d476
> < openapi-schema-validator==0.6.3
> < openapi-spec-validator==0.7.2
> 560d506
> < pathable==0.4.4
> 563d508
> < pdbr==0.9.2
> 569,570c514
> < pinotdb==5.6.0
> < pipdeptree==2.28.0
> ---
> > pinotdb==5.7.0
> 574d517
> < plyvel==1.5.1
> 577,579d519
> < pprintpp==0.4.0
> < pre-commit-uv==4.1.4
> < pre_commit==4.2.0
> 592d531
> < py-partiql-parser==0.6.1
> 608d546
> < pyenchant==3.2.2
> 610,611d547
> < pygraphviz==1.14
> < pyiceberg==0.9.1
> 622,632d557
> < pytest-asyncio==0.25.0
> < pytest-cov==6.2.1
> < pytest-custom-exit-code==0.3.0
> < pytest-icdiff==0.9
> < pytest-instafail==0.5.0
> < pytest-mock==3.14.1
> < pytest-rerunfailures==15.1
> < pytest-timeouts==1.2.1
> < pytest-unordered==0.7.0
> < pytest-xdist==3.8.0
> < pytest==8.4.1
> 642d566
> < python-on-whales==0.78.0
> 652d575
> < readme_renderer==44.0
> 659d581
> < requests-mock==1.12.1
> 664,665d585
> < responses==0.25.7
> < restructuredtext_lint==1.4.0
> 667,668d586
> < rfc3339-validator==0.1.4
> < rfc3986==2.0.0
> 670d587
> < rich-click==1.8.9
> 674d590
> < roman-numerals-py==3.1.0
> 679d594
> < ruff==0.12.3
> 688d602
> < semver==3.0.4
> 690,691c604
> < sentinels==1.0.0
> < sentry-sdk==2.33.1
> ---
> > sentry-sdk==2.33.2
> 703d615
> < snowballstemmer==3.0.1
> 709,725d620
> < sphinx-argparse==0.5.2
> < sphinx-autoapi==3.6.0
> < sphinx-autobuild==2024.10.3
> < sphinx-copybutton==0.5.2
> < sphinx-jinja==2.0.2
> < sphinx-rtd-theme==3.0.2
> < sphinx_design==0.6.1
> < sphinxcontrib-applehelp==2.0.0
> < sphinxcontrib-devhelp==2.0.0
> < sphinxcontrib-htmlhelp==2.1.0
> < sphinxcontrib-httpdomain==1.8.1
> < sphinxcontrib-jquery==4.1
> < sphinxcontrib-jsmath==1.0.1
> < sphinxcontrib-qthelp==2.0.0
> < sphinxcontrib-redoc==1.6.0
> < sphinxcontrib-serializinghtml==2.0.0
> < sphinxcontrib-spelling==8.0.1
> 737d631
> < strictyaml==1.7.3
> 740d633
> < sympy==1.14.0
> 752d644
> < time-machine==2.16.0
> 755d646
> < tomli_w==1.2.0
> 758d648
> < towncrier==24.8.0
> 762,763d651
> < trove-classifiers==2025.5.9.12
> < twine==6.1.0
> 765,774d652
> < types-Deprecated==1.2.15.20250304
> < types-Markdown==3.8.0.20250708
> < types-PyMySQL==1.1.0.20250711
> < types-PyYAML==6.0.12.20250516
> < types-aiofiles==24.1.0.20250708
> < types-certifi==2021.10.8.3
> < types-cffi==1.17.0.20250523
> < types-croniter==6.0.0.20250626
> < types-docutils==0.21.0.20250722
> < types-paramiko==3.5.0.20250708
> 776,778d653
> < types-pyOpenSSL==24.1.0.20240722
> < types-python-dateutil==2.9.0.20250708
> < types-python-slugify==8.0.2.20240310
> 780d654
> < types-redis==4.6.0.20241004
> 782,784d655
> < types-setuptools==80.9.0.20250529
> < types-tabulate==0.9.0.20241207
> < types-toml==0.10.8.20240310
> 794d664
> < userpath==1.9.2
> 799c669
> < validators==0.34.0
> ---
> > validators==0.35.0
> 806c676
> < weaviate-client==4.9.6
> ---
> > weaviate-client==4.16.3
> 809c679
> < websockets==14.2
> ---
> > websockets==15.0.1
> 815d684
> < yamllint==1.37.1
> ________________________________
> Strike Technologies, LLC (“Strike”) is part of the GTS family of
> companies. Strike is a technology solutions provider, and is not a broker
> or dealer and does not transact any securities related business directly
> whatsoever. This communication is the property of Strike and its
> affiliates, and does not constitute an offer to sell or the solicitation of
> an offer to buy any security in any jurisdiction. It is intended only for
> the person to whom it is addressed and may contain information that is
> privileged, confidential, or otherwise protected from disclosure.
> Distribution or copying of this communication, or the information contained
> herein, by anyone other than the intended recipient is prohibited. If you
> have received this communication in error, please immediately notify Strike
> at i...@striketechnologies.com, and delete and destroy any copies hereof.
> ________________________________
>
> CONFIDENTIALITY / PRIVILEGE NOTICE: This transmission and any attachments
> are intended solely for the addressee. This transmission is covered by the
> Electronic Communications Privacy Act, 18 U.S.C ''2510-2521. The
> information contained in this transmission is confidential in nature and
> protected from further use or disclosure under U.S. Pub. L. 106-102, 113
> U.S. Stat. 1338 (1999), and may be subject to attorney-client or other
> legal privilege. Your use or disclosure of this information for any purpose
> other than that intended by its transmittal is strictly prohibited, and may
> subject you to fines and/or penalties under federal and state law. If you
> are not the intended recipient of this transmission, please DESTROY ALL
> COPIES RECEIVED and confirm destruction to the sender via return
> transmittal.
>
