> +1 to this idea. I considered whether we should use `pylock.toml` instead, but it might not address the issue we intend to resolve based on my reading.
Yeah. pylock.toml and https://peps.python.org/pep-0751/ (in the future) are better replacement for point 3: > 3) to give our users a way to install released airflow in reproducible way This is the intention of pylock.toml - it should eventually allow our users to install airflow in a reproducible way so basically would replace the constraints of ours. For now support for it is experimental in PyPI and it might change, also there is a general consensus that it does not fully replace the ".lock" files feature for poetry, uv, pip-tools when it comes to local development, workspaces, monorepos (which don't even have standardisation yet). Also `uv` supports PEP751 pylock.toml as a (loosy - it removes a number of metada kept in uv.lock specific for development workflows) export format from its uv.lock. My view (and also a lot of the information I got from packaging people i interact with) - that each of the "development/project management" tools (like uv/poetry etc.) will keep their own ".lock" format - simply because there is no possible way to encompass all the needs they have in "generic way". There was about 100 os pages discussion when PEP 751 was proposed and uv team took a very active part in it - but there seem to be a number of points that are not really "generic". For example pip is not aiming and will likely never be a "development/project management" tool and it does not have a "workspace project" - it will always be really "end-user installatiion tool" as it's primary function - and the result of this different approach is what PEP 751 aims for - to allow reproducible installation for end users. So what I see as a natural step when we adopt `uv.lock` as development "source of truth" and our "development/project management tool" - we will be able to use `uv` export to pylock.toml and replace our `--constraint` installation. But while several tools (both pip and uv) already support exporting to the pylock.toml, there is no way YET to install packages using such lock to actually install packages. Pip for example has this issue open for it https://github.com/pypa/pip/issues/13334 and it waits for someone to volunteer and implement it. So for us it would make sense to switch for 3) from constraints to pylock.toml - but only after `pip` and other way people install airflow will be able to use it. Before that - there is no benefit of us using it. This might even stay like that: `uv.lock` for development and pylock.toml published by us for users who want to install airflow in reproducible way. The `pylock.toml` has a number of benefits comparing to current constraint approach - single lock supporting multiple python versions, cryptographic integrity and attestation support, support for multiple platforms, support for dependency groups, support for default groups - all that will make few of our "custom" mechanisms we currently "workaround" into the "solid features" of reproducible installation. But we are not there yet :).\ J. On Tue, Aug 19, 2025 at 10:59 AM Wei Lee <weilee...@gmail.com> wrote: > +1 to this idea. I considered whether we should use `pylock.toml` instead, > but it might not address the issue we intend to resolve based on my reading. > > Best, > Wei > > > On Aug 19, 2025, at 5:51 AM, Ferruzzi, Dennis > <ferru...@amazon.com.INVALID> wrote: > > > > I don't have any particular experience, but based just on what you > shared there, it does feel like a next logical step in the migration to > these tools. Off-loading work and simplifying our CI is also always a plus > in my opinion. > > > > > > - ferruzzi > > > > > > ________________________________ > > From: Jarek Potiuk <ja...@potiuk.com> > > Sent: Monday, August 18, 2025 4:14 AM > > To: dev@airflow.apache.org > > Subject: [EXT] [DISCUSS] Replace constraints with uv.lock mechanisms for > dev env freeze > > > > CAUTION: This email originated from outside of the organization. Do not > click links or open attachments unless you can confirm the sender and know > the content is safe. > > > > > > > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur > externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous > ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas > certain que le contenu ne présente aucun risque. > > > > > > > > Hello here, > > > > *TL;DR: I would love to hear comments on the proposal to switch to using > > uv.lock for our dev env dependency freeze* > > > > While working on some small follow-ups after prek upgrade, I think I > > finally figured out how we can start using `uv.lock` locally and merge it > > into our constraint mechanism effectively - without > > over-burdening committers with merging a lot of "dependabot-like" PRs. > > > > The key problem that needed to be resolved was who and when would update > > the uv.lock regularly. > > > > I captured my thoughts in https://github.com/apache/airflow/issues/54609 > > > > But let me copy the content of it below. I would love to hear the > comments > > - here or in the GitHub issue if you will (I will bring important things > > from the issue here if they will be raised there): > > > > > > *Currently constraints are used for three purposes:* > > 1) to take snapshot of latest "working" version of dependencies > > 2) to limit PRs to "latest known good constraints" - so that newly > released > > packages are not failing most PRs, but fail canary builds instead > > 3) to give our users a way to install released airflow in reproducible > way > > > > However, after we switched to uv, the case 2) can be better achieved with > > uv.lock. Currently we .gitignore uv.lock, but we could commit it to the > > repository and handle constraint generation differently > > > > * constraints could be generated using latest lock > > * canary builds could use the --resolution highest as today and diverge > > from latest "frozen" uv.lock > > * all prs would use the uv.lock and do not use --resolution-highest in > > those PRs. This has the added benefit that uv sync in PR would > > automatically upgrade dependencies if the dependencies in pyproject.toml > > are conflicting with uv.lock - but it would not use --resolution highest, > > which means that the PRs that modify dependeNcies do not have to get > latest > > dependencies when they are run - that would limit the number of PRs that > > are experiencing "main/canary" problems > > > > *Problem* > > > > The only problem to solve is who and when would upgrade the uv.lock and > > commit it to the repo. Currently constraints are automatically bumped > after > > successful canary runs, but they are in orphan un-protected branches, > while > > uv.lock shares the code with main airflow repo, and we need a "human with > > ICLA" to review and merge commits there (ASF requirement) > > > > > > *Solution* > > Proposed solution is to add such a uv.lock upgrade to our > > "upgrade-important-versions" prek hook. We run it in CI in canary builds > > and we fail the builds when new versions of important installers change > > (pip, Python base images, golang) - this happens often-enough to be > > eventually-consistent pretty fast, but rarely enough to make it into > > annoying "everyday" upgrade chore. > > > > Regular PRs - especially those that would change dependencies will also > > modify uv.lock - for example when a new dependency is added or removed or > > bumped. This means that the uv.lock will be generally quite regularly > > bumped and "eventually consistent" with constraints. > > > > > > *Benefits:* > > * simplification of breeze build image with constraints (currently it > uses > > complex uv pip install with --constraint, and it could simply use uv sync > > --all-packages instead if our uv.lock is checked in and maintained > > * better synchronization of the local venv. Currently uv sync does not > > guarantee to have the same version of dependencies in local venv. The > only > > way to guarantee it now is to use breeze image - build with constraints > or > > run complex commands with constraints added when you run uv sync > > * less "3rd-party release" impact on failing PRs. if PRs will not touch > the > > dependency specification where the new release is released, it will > > continue using the dependency stored in .lock and not the latest > dependency > > released > > > > > > *Cons:* > > * Probably we should pay more attention to run upgrade-important-versions > > regularly and even add an option there to manually just perform the sync > > (for example that will be useful as part of the release procedure, where > > release manager would have to make sure to run the > > upgrade-important-versions check when preparing release notes. But it has > > the advantage that it will be a conscious step that should actually help > in > > making sure that constraints are updated. > > > > * sometimes PRs of people will have "big" changes to uv.lock - when they > > are updating the dependency and alongside the uv.lock will also bump many > > other unrelated dependencies - this could be mitigated by splitting it > into > > "lock upgrade" and "dependency update" but workflow of that is not very > > convenient and natural. > > > > J. > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > For additional commands, e-mail: dev-h...@airflow.apache.org > >
