Would love to hear more if anyone has an opinion :). Yep. It's an advanced
topic and I can make arbitrary decision :) ...
I am thinking now that it should be quite "OK" to accept those "potentially
untested" versions - especially that they will be tested during manual
testing. Unless someone objects i will likely update PR with some changelog
/ newsfragment to announce the change in 3.1. I will run LAZY CONSENSUS
soon if I don't hear any opposition.
J.
On Tue, Jul 22, 2025 at 5:52 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> > Exactly to remove the dev requirements airflow has from interfering with
> our own dev requirements. Though this approach notably doesn't change any
> of the used non-dev requirements, it's not clear to me (and I'm probably
> just missing something) why your diff includes non-dev requirements.
>
> The thing is that I do not "remove" devel features - because due to
> transitive dependencies I would have to go deep down the rabbit hole which
> dependencies are "devel" and which not. So what I do, I ask `uv` to do the
> job for me and repeat the resolution but with the `--exact` flag (combined
> with the already used `--resolution higest` and `--no-sources`).
>
> Apparently in this case `uv` will sometimes decide that it's ok to bump
> some dependencies that are left after initial resolution. The thing is that
> uv will not install some versions of some packages if they are conflicting
> with already installed packages. What I believe `uv` does when I use the
> *--exact
> *flag of `uv pip install` it "fixes" the initial resolution. It finds out
> what is needed, removes those packages that are not needed, and then it
> resolves again and improves the resolution (that would be my guess at least
> - it could be done slightly differently and in a single step I guess.
>
> Essentially the difference between the left and right side is:
>
> a) install absolutely everything from the workspace
>
> b) do this:
> left: uv pip install --no-sources 'apache-airflow[all]'
> 'apache-airflow-core[all]' apache-airflow-task-sdk ./airflow-ctl
> --reinstall --resolution highest --find-links file:///dist
> right: uv pip install --no-sources *--exact --strict*
> 'apache-airflow[all]' 'apache-airflow-core[all]' apache-airflow-task-sdk
> ./airflow-ctl --reinstall --resolution highest --find-links file:///dist
>
> In the current state content of the fille://dist should be the same in
> both cases:
>
> apache-airflow-providers-edge3 (1.1.2): wheel -
> apache_airflow_providers_edge3-1.1.2-py3-none-any.whl
> apache-airflow-providers-keycloak (1.0.0): wheel -
> apache_airflow_providers_keycloak-1.0.0-py3-none-any.whl
> apache-airflow-providers-snowflake (6.5.1): wheel -
> apache_airflow_providers_snowflake-6.5.1-py3-none-any.whl
>
> This is a bit of in-process quirk of the "PyPI" resolution - those are the
> three packages that we have updated versions of in the sources, but they
> are not released yet. But they should be essentially the same in both
> cases. So the difference between left and right is `--exact --strict`
> (strict I added to make sure that we are not removing something that is
> still needed).
>
> J.
>
>
>
>
>
> On Tue, Jul 22, 2025 at 5:22 PM Damian Shaw <ds...@striketechnologies.com>
> wrote:
>
>> As I user of Airflow I have ended up implementing this for my
>> environments:
>>
>> pip install apache-airflow[all or select-extras] -c {official
>> constraints} > smaller-run-time-constraints.txt
>> pip-compile {all my requirements including development}.txt -c
>> smaller-run-time-constraints.txt
>>
>> Exactly to remove the dev requirements airflow has from interfering with
>> our own dev requirements. Though this approach notably doesn't change any
>> of the used non-dev requirements, it's not clear to me (and I'm probably
>> just missing something) why your diff includes non-dev requirements.
>>
>> Damian
>>
>> -----Original Message-----
>> From: Jarek Potiuk <ja...@potiuk.com>
>> Sent: Tuesday, July 22, 2025 11:04 AM
>> To: dev@airflow.apache.org
>> Subject: [DISCUSS] Removing dev dependencies from PyPI constraints ?
>>
>> Hello here,
>>
>>
>> *Tl;DR; We had an interesting discussion on Slack today and I wanted to
>> bring it here to discuss what we should do - whether we should keep devel
>> dependencies in our PyPI constraints or not.*
>>
>> *A bit of context:*
>> In our CI we generate several types of constraints. one of them are
>> "source" constraints that we use during our CI testing - including all our
>> testing tools, mypy, ruff etc., and another type is "PyPI" constraint -
>> which is more <What should be the "golden" versions of dependencies when
>> you install Airflow using `pip install` for production>.
>>
>> The main difference between the two (roughly) is that "source constraints"
>> only uses dependencies from "branch tip" ("main", "v3-0-test"), where
>> "PyPI constraints" uses "Airflow core" branch-tip dependencies - but all
>> the providers are installed from PyPI. There are few nuances when some
>> packages are being released but basically - for example for Airflow 3.0.3,
>> "PyPI"
>> constraints is "what is the current set of dependencies and providers
>> when we install airflow from PyPI at the moment of release".
>>
>> So target for "source" constraints are "Airflow contributors" where
>> target for "PyPI constraints" are "Airflow users".
>>
>>
>> *The problem*
>> In the slack discussion
>> https://apache-airflow.slack.com/archives/C06K9Q5G2UA/p1753169914722049
>> - slack user @dashmug (sorry - do not know name) noticed that the "PyPI"
>> constraints also contain Airflow development dependencies listed (like
>> ruff or mypy). That made me think a bit and I quickly came up with PR that
>> could "fix" it - i.e. only keep the "real" constraints in.
>> https://github.com/apache/airflow/pull/53631 - it's as easy as adding
>> the `--exact` (and `--strict` for verification) flag to the `uv pip
>> install` command we use.
>>
>> I agree it's a bit confusing to have the devel dependencies there - it is
>> not the "intention", but there is very little problem with it - the fact
>> that mypy or ruff are listed with a version in constraints, does not mean
>> it must be installed - constraints mainly say "if you install this, it
>> should be this version" . And there is no problem whatsoever (and it is
>> even recommended by us) to have "airflow with its deps" installed with
>> constraints, and then anything else you need to install or update -
>> without. So other than potential confusion, it usually does not cause harm.
>>
>> Now... It might seem that removing devel deps has no side effects and we
>> should **just do it** - but it's not that easy. Main problem is that those
>> devel dependencies might "hold" other dependencies (directly or
>> transitively) from being upgraded - and when we remove the devel
>> dependencies and do the resolution again - we might have a DIFFERENT set of
>> dependencies, simply because removing the devel dependencies, might "free"
>> the other deps and they might get upgraded. Yeah - I know it's not
>> obvious for someone who did not spend 3 years solving dependency issues :).
>>
>> And the problem with it is that because we need those devel deps to test
>> things - the upgraded dependencies might have never been tested in our CI -
>> because we never upgraded to later versions. It's a bit of
>> Heisenberg-effect. By using test tools we are changing the state of the
>> tested thing. Currently our "PyPI" constraints reflect the state of Airflow
>> "when the testing tools are installed" - and we are generally not able to
>> test the state of Airflow "without the testing tools" - by the sheer fact
>> that we do not have those tools :)
>>
>> Just to illustrate the case - you can see what happens when we removed
>> devel deps:
>> https://github.com/apache/airflow/actions/runs/16444553234 - in summary
>> you can see a colored version that is a bit clearer but I dumped the diff
>> below from an example - Python 3.10 case. Left side < is "what was with
>> testing tools" and > is without them. You can see a LOT of test tools
>> removed (pytest, ruff and others - including dependencies used only by
>> those tools)
>>
>> But also you can see (likely some of the devel/test deps limited those)
>>
>> * Authlib - upgraded from 1.3.1 to 1.6.1
>> * Google - genai upgraded from 1.2.0 to 1.20.0
>> * Httpx - upgraded from 0.27.0 to 0.28.1
>> * Pinotdb - upgraded from 5.6.0 to 5.7.0
>> * Sentry-sdk - upgraded from 2.33.1 to 2.33.2 (this is false positive as
>> it has just been released and new version was picked)
>> * validators - upgraded form 0.34.0 to 0.35.0
>> * weaviate-client - upgraded from 4.9.0 to 4.16.3
>> * websockets - upgraded from 14.2 to 15.0.1
>>
>>
>> This means that all the "upgraded" packages above have not been - likely
>> ever tested in our CI.
>>
>> And when we put them in "versioned" constraints, it might mean that they
>> will be obviously failing.
>>
>> So ... there is a risk connected with just generating the "non-devel"
>> constraints without additional tests. But that also means that the user
>> who does not use constraints might get those versions anyway - default
>> behaviour of `pip install airflow` will be to install those upgraded
>> versions above if you do not use constraints. But also that is no different
>> than installing a released airflow version a few weeks or months after
>> release without constraints - there will be tens of packages that were
>> released since our release and installing airflow without constraints will
>> pick those.
>>
>> However - when we manually install RC candidates - say by running airflow
>> from rc1 image, we will use those "upgraded" dependencies - so some
>> "obvious" problems might be caught during the rc manual testing - where you
>> do not use test tools but run airflow "as if" it was installed in
>> production.
>>
>> *The solution ? *
>>
>> There are few ways we can proceed with it:
>>
>> a) leave it as is - no big harm, potential confusion, by people who see
>> it
>> - potential differences
>>
>> b) remove devel following my PR and take the risk that we never tested (in
>> CI) the dependencies we set in the constraints. That's a bit risky, but
>> we can always reactively downgrade some of those packages after the fact if
>> users will start reporting the issues
>>
>> c) potentially most complex thing - trying to remove devel dependencies
>> somewhat combining the two - for example removing all the deps that would
>> be removed in b) from a) - but that has also some edge cases - when
>> different versions of dependencies can add or upgrade their dependencies,
>> we are very likely to occasionally remove or add too much.
>>
>>
>> I wonder (for those who managed to read that far - likely not many - what
>> do you think here? What would be the best option ? Maybe there are other
>> options I have not thought about ?
>>
>>
>> J
>>
>>
>> -----
>>
>> The diff for Python 3.10
>>
>>
>>
>> 34c34
>> < Authlib==1.3.1
>> ---
>> > Authlib==1.6.1
>> 64d63
>> < Sphinx==8.2.3
>> 78d76
>> < aioresponses==0.7.8
>> 82d79
>> < alabaster==1.0.0
>> 198d194
>> < arro3-core==0.5.1
>> 202d197
>> < astroid==3.3.11
>> 211,212d205
>> < aws-sam-translator==1.99.0
>> < aws-xray-sdk==2.14.0
>> 258,259d250
>> < cfgv==3.4.0
>> < cfn-lint==1.38.0
>> 262d252
>> < checksumdir==1.2.0
>> 277d266
>> < coverage==7.9.2
>> 291d279
>> < deltalake==1.1.0
>> 293d280
>> < diagrams==0.24.4
>> 301,302d287
>> < docutils==0.21.2
>> < duckdb==1.3.2
>> 308d292
>> < eralchemy2==1.4.1
>> 311d294
>> < execnet==2.1.1
>> 321,322d303
>> < flit==3.12.0
>> < flit_core==3.12.0
>> 386c367
>> < google-genai==1.2.0
>> ---
>> > google-genai==1.20.0
>> 389d369
>> < graphql-core==3.2.6
>> 398d377
>> < grpcio-tools==1.62.3
>> 403,404d381
>> < hatch==1.14.1
>> < hatchling==1.27.0
>> 413c390
>> < httpx==0.27.0
>> ---
>> > httpx==0.28.1
>> 418d394
>> < hyperlink==21.0.0
>> 421,423d396
>> < icdiff==2.0.7
>> < id==1.5.0
>> < identify==2.6.12
>> 426d398
>> < imagesize==1.4.1
>> 430d401
>> < incremental==24.7.2
>> 433,435d403
>> < iniconfig==2.1.0
>> < inputimeout==1.0.4
>> < ipdb==0.13.13
>> 449d416
>> < joserfc==1.2.2
>> 451d417
>> < jsonpatch==1.33
>> 455,456d420
>> < jsonpointer==3.0.0
>> < jsonschema-path==0.3.4
>> 463d426
>> < kerberos==1.3.1
>> 465,466d427
>> < keyrings.alt==5.0.2
>> < kgb==7.2
>> 494,495d454
>> < mmh3==5.1.0
>> < mongomock==4.3.0
>> 497,498d455
>> < moto==5.1.8
>> < mpmath==1.3.0
>> 508,512d464
>> < mypy-boto3-appflow==1.39.0
>> < mypy-boto3-rds==1.39.1
>> < mypy-boto3-redshift-data==1.39.0
>> < mypy-boto3-s3==1.39.5
>> < mypy==1.17.0
>> 514c466
>> < mysql-connector-python==9.3.0
>> ---
>> > mysql-connector-python==9.4.0
>> 521,523d472
>> < networkx==3.5
>> < nh3==0.3.0
>> < nodeenv==1.9.1
>> 528,529d476
>> < openapi-schema-validator==0.6.3
>> < openapi-spec-validator==0.7.2
>> 560d506
>> < pathable==0.4.4
>> 563d508
>> < pdbr==0.9.2
>> 569,570c514
>> < pinotdb==5.6.0
>> < pipdeptree==2.28.0
>> ---
>> > pinotdb==5.7.0
>> 574d517
>> < plyvel==1.5.1
>> 577,579d519
>> < pprintpp==0.4.0
>> < pre-commit-uv==4.1.4
>> < pre_commit==4.2.0
>> 592d531
>> < py-partiql-parser==0.6.1
>> 608d546
>> < pyenchant==3.2.2
>> 610,611d547
>> < pygraphviz==1.14
>> < pyiceberg==0.9.1
>> 622,632d557
>> < pytest-asyncio==0.25.0
>> < pytest-cov==6.2.1
>> < pytest-custom-exit-code==0.3.0
>> < pytest-icdiff==0.9
>> < pytest-instafail==0.5.0
>> < pytest-mock==3.14.1
>> < pytest-rerunfailures==15.1
>> < pytest-timeouts==1.2.1
>> < pytest-unordered==0.7.0
>> < pytest-xdist==3.8.0
>> < pytest==8.4.1
>> 642d566
>> < python-on-whales==0.78.0
>> 652d575
>> < readme_renderer==44.0
>> 659d581
>> < requests-mock==1.12.1
>> 664,665d585
>> < responses==0.25.7
>> < restructuredtext_lint==1.4.0
>> 667,668d586
>> < rfc3339-validator==0.1.4
>> < rfc3986==2.0.0
>> 670d587
>> < rich-click==1.8.9
>> 674d590
>> < roman-numerals-py==3.1.0
>> 679d594
>> < ruff==0.12.3
>> 688d602
>> < semver==3.0.4
>> 690,691c604
>> < sentinels==1.0.0
>> < sentry-sdk==2.33.1
>> ---
>> > sentry-sdk==2.33.2
>> 703d615
>> < snowballstemmer==3.0.1
>> 709,725d620
>> < sphinx-argparse==0.5.2
>> < sphinx-autoapi==3.6.0
>> < sphinx-autobuild==2024.10.3
>> < sphinx-copybutton==0.5.2
>> < sphinx-jinja==2.0.2
>> < sphinx-rtd-theme==3.0.2
>> < sphinx_design==0.6.1
>> < sphinxcontrib-applehelp==2.0.0
>> < sphinxcontrib-devhelp==2.0.0
>> < sphinxcontrib-htmlhelp==2.1.0
>> < sphinxcontrib-httpdomain==1.8.1
>> < sphinxcontrib-jquery==4.1
>> < sphinxcontrib-jsmath==1.0.1
>> < sphinxcontrib-qthelp==2.0.0
>> < sphinxcontrib-redoc==1.6.0
>> < sphinxcontrib-serializinghtml==2.0.0
>> < sphinxcontrib-spelling==8.0.1
>> 737d631
>> < strictyaml==1.7.3
>> 740d633
>> < sympy==1.14.0
>> 752d644
>> < time-machine==2.16.0
>> 755d646
>> < tomli_w==1.2.0
>> 758d648
>> < towncrier==24.8.0
>> 762,763d651
>> < trove-classifiers==2025.5.9.12
>> < twine==6.1.0
>> 765,774d652
>> < types-Deprecated==1.2.15.20250304
>> < types-Markdown==3.8.0.20250708
>> < types-PyMySQL==1.1.0.20250711
>> < types-PyYAML==6.0.12.20250516
>> < types-aiofiles==24.1.0.20250708
>> < types-certifi==2021.10.8.3
>> < types-cffi==1.17.0.20250523
>> < types-croniter==6.0.0.20250626
>> < types-docutils==0.21.0.20250722
>> < types-paramiko==3.5.0.20250708
>> 776,778d653
>> < types-pyOpenSSL==24.1.0.20240722
>> < types-python-dateutil==2.9.0.20250708
>> < types-python-slugify==8.0.2.20240310
>> 780d654
>> < types-redis==4.6.0.20241004
>> 782,784d655
>> < types-setuptools==80.9.0.20250529
>> < types-tabulate==0.9.0.20241207
>> < types-toml==0.10.8.20240310
>> 794d664
>> < userpath==1.9.2
>> 799c669
>> < validators==0.34.0
>> ---
>> > validators==0.35.0
>> 806c676
>> < weaviate-client==4.9.6
>> ---
>> > weaviate-client==4.16.3
>> 809c679
>> < websockets==14.2
>> ---
>> > websockets==15.0.1
>> 815d684
>> < yamllint==1.37.1
>> ________________________________
>> Strike Technologies, LLC (“Strike”) is part of the GTS family of
>> companies. Strike is a technology solutions provider, and is not a broker
>> or dealer and does not transact any securities related business directly
>> whatsoever. This communication is the property of Strike and its
>> affiliates, and does not constitute an offer to sell or the solicitation of
>> an offer to buy any security in any jurisdiction. It is intended only for
>> the person to whom it is addressed and may contain information that is
>> privileged, confidential, or otherwise protected from disclosure.
>> Distribution or copying of this communication, or the information contained
>> herein, by anyone other than the intended recipient is prohibited. If you
>> have received this communication in error, please immediately notify Strike
>> at i...@striketechnologies.com, and delete and destroy any copies hereof.
>> ________________________________
>>
>> CONFIDENTIALITY / PRIVILEGE NOTICE: This transmission and any attachments
>> are intended solely for the addressee. This transmission is covered by the
>> Electronic Communications Privacy Act, 18 U.S.C ''2510-2521. The
>> information contained in this transmission is confidential in nature and
>> protected from further use or disclosure under U.S. Pub. L. 106-102, 113
>> U.S. Stat. 1338 (1999), and may be subject to attorney-client or other
>> legal privilege. Your use or disclosure of this information for any purpose
>> other than that intended by its transmittal is strictly prohibited, and may
>> subject you to fines and/or penalties under federal and state law. If you
>> are not the intended recipient of this transmission, please DESTROY ALL
>> COPIES RECEIVED and confirm destruction to the sender via return
>> transmittal.
>>
>
