Thanks Jarek for bringing this up, I really like this idea we should use uv.lock.
For upgrading packages in the uv.lock file, dependaboat started adding support for this, but it seems there are quite a lot of issues. Some i have seen dependaboat updates lock file but not updating the package in toml file etc; there may be other issues similar to related. but I hope these may be addressed at some point of time ( Not sure when :) ) https://docs.astral.sh/uv/guides/integration/dependency-bots/#dependabot https://github.com/astral-sh/uv/issues/2512 https://github.com/dependabot/dependabot-core/issues?q=is%3Aissue%20state%3Aopen%20uv Pavan On Tue, Aug 19, 2025 at 11:18 AM Wei Lee <weilee...@gmail.com> wrote: > > pip is not aiming and > > will likely never be a "development/project management" tool and > > I’m a bit surprised. I thought the dep group thing was for this. > > >> 3) to give our users a way to install released airflow in reproducible > way > > > be able to use `uv` export to pylock.toml and replace our `--constraint` > > installation. > > I’m thinking of supporting both. Or at least for a while, till > `pylock.toml` becomes well-known (which might take a long long time…) > They’re generated files, so might not be that hard to maintain. > > > Best, > Wei > > > On Aug 19, 2025, at 5:45 PM, Jarek Potiuk <ja...@potiuk.com> wrote: > > > >> +1 to this idea. I considered whether we should use `pylock.toml` > > instead, but it might not address the issue we intend to resolve based on > > my reading. > > > > Yeah. pylock.toml and https://peps.python.org/pep-0751/ (in the future) > are > > better replacement for point 3: > > > >> 3) to give our users a way to install released airflow in reproducible > way > > > > This is the intention of pylock.toml - it should eventually allow our > users > > to install airflow in a reproducible way so basically would replace the > > constraints of ours. For now support for it is experimental in PyPI and > it > > might change, also there is a general consensus that it does not fully > > replace the ".lock" files feature for poetry, uv, pip-tools when it comes > > to local development, workspaces, monorepos (which don't even have > > standardisation yet). Also `uv` supports PEP751 pylock.toml as a (loosy - > > it removes a number of metada kept in uv.lock specific for development > > workflows) export format from its uv.lock. > > > > My view (and also a lot of the information I got from packaging people i > > interact with) - that each of the "development/project management" tools > > (like uv/poetry etc.) will keep their own ".lock" format - simply because > > there is no possible way to encompass all the needs they have in "generic > > way". There was about 100 os pages discussion when PEP 751 was proposed > and > > uv team took a very active part in it - but there seem to be a number of > > points that are not really "generic". For example pip is not aiming and > > will likely never be a "development/project management" tool and it does > > not have a "workspace project" - it will always be really "end-user > > installatiion tool" as it's primary function - and the result of this > > different approach is what PEP 751 aims for - to allow reproducible > > installation for end users. > > > > So what I see as a natural step when we adopt `uv.lock` as development > > "source of truth" and our "development/project management tool" - we will > > be able to use `uv` export to pylock.toml and replace our `--constraint` > > installation. But while several tools (both pip and uv) already support > > exporting to the pylock.toml, there is no way YET to install packages > using > > such lock to actually install packages. Pip for example has this issue > open > > for it https://github.com/pypa/pip/issues/13334 and it waits for > someone > > to volunteer and implement it. So for us it would make sense to switch > for > > 3) from constraints to pylock.toml - but only after `pip` and other way > > people install airflow will be able to use it. Before that - there is no > > benefit of us using it. This might even stay like that: `uv.lock` for > > development and pylock.toml published by us for users who want to install > > airflow in reproducible way. The `pylock.toml` has a number of benefits > > comparing to current constraint approach - single lock supporting > multiple > > python versions, cryptographic integrity and attestation support, support > > for multiple platforms, support for dependency groups, support for > default > > groups - all that will make few of our "custom" mechanisms we currently > > "workaround" into the "solid features" of reproducible installation. But > we > > are not there yet :).\ > > > > J. > > > > > > On Tue, Aug 19, 2025 at 10:59 AM Wei Lee <weilee...@gmail.com> wrote: > > > >> +1 to this idea. I considered whether we should use `pylock.toml` > instead, > >> but it might not address the issue we intend to resolve based on my > reading. > >> > >> Best, > >> Wei > >> > >>> On Aug 19, 2025, at 5:51 AM, Ferruzzi, Dennis > >> <ferru...@amazon.com.INVALID> wrote: > >>> > >>> I don't have any particular experience, but based just on what you > >> shared there, it does feel like a next logical step in the migration to > >> these tools. Off-loading work and simplifying our CI is also always a > plus > >> in my opinion. > >>> > >>> > >>> - ferruzzi > >>> > >>> > >>> ________________________________ > >>> From: Jarek Potiuk <ja...@potiuk.com> > >>> Sent: Monday, August 18, 2025 4:14 AM > >>> To: dev@airflow.apache.org > >>> Subject: [EXT] [DISCUSS] Replace constraints with uv.lock mechanisms > for > >> dev env freeze > >>> > >>> CAUTION: This email originated from outside of the organization. Do not > >> click links or open attachments unless you can confirm the sender and > know > >> the content is safe. > >>> > >>> > >>> > >>> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur > >> externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si > vous > >> ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas > >> certain que le contenu ne présente aucun risque. > >>> > >>> > >>> > >>> Hello here, > >>> > >>> *TL;DR: I would love to hear comments on the proposal to switch to > using > >>> uv.lock for our dev env dependency freeze* > >>> > >>> While working on some small follow-ups after prek upgrade, I think I > >>> finally figured out how we can start using `uv.lock` locally and merge > it > >>> into our constraint mechanism effectively - without > >>> over-burdening committers with merging a lot of "dependabot-like" PRs. > >>> > >>> The key problem that needed to be resolved was who and when would > update > >>> the uv.lock regularly. > >>> > >>> I captured my thoughts in > https://github.com/apache/airflow/issues/54609 > >>> > >>> But let me copy the content of it below. I would love to hear the > >> comments > >>> - here or in the GitHub issue if you will (I will bring important > things > >>> from the issue here if they will be raised there): > >>> > >>> > >>> *Currently constraints are used for three purposes:* > >>> 1) to take snapshot of latest "working" version of dependencies > >>> 2) to limit PRs to "latest known good constraints" - so that newly > >> released > >>> packages are not failing most PRs, but fail canary builds instead > >>> 3) to give our users a way to install released airflow in reproducible > >> way > >>> > >>> However, after we switched to uv, the case 2) can be better achieved > with > >>> uv.lock. Currently we .gitignore uv.lock, but we could commit it to the > >>> repository and handle constraint generation differently > >>> > >>> * constraints could be generated using latest lock > >>> * canary builds could use the --resolution highest as today and diverge > >>> from latest "frozen" uv.lock > >>> * all prs would use the uv.lock and do not use --resolution-highest in > >>> those PRs. This has the added benefit that uv sync in PR would > >>> automatically upgrade dependencies if the dependencies in > pyproject.toml > >>> are conflicting with uv.lock - but it would not use --resolution > highest, > >>> which means that the PRs that modify dependeNcies do not have to get > >> latest > >>> dependencies when they are run - that would limit the number of PRs > that > >>> are experiencing "main/canary" problems > >>> > >>> *Problem* > >>> > >>> The only problem to solve is who and when would upgrade the uv.lock and > >>> commit it to the repo. Currently constraints are automatically bumped > >> after > >>> successful canary runs, but they are in orphan un-protected branches, > >> while > >>> uv.lock shares the code with main airflow repo, and we need a "human > with > >>> ICLA" to review and merge commits there (ASF requirement) > >>> > >>> > >>> *Solution* > >>> Proposed solution is to add such a uv.lock upgrade to our > >>> "upgrade-important-versions" prek hook. We run it in CI in canary > builds > >>> and we fail the builds when new versions of important installers change > >>> (pip, Python base images, golang) - this happens often-enough to be > >>> eventually-consistent pretty fast, but rarely enough to make it into > >>> annoying "everyday" upgrade chore. > >>> > >>> Regular PRs - especially those that would change dependencies will also > >>> modify uv.lock - for example when a new dependency is added or removed > or > >>> bumped. This means that the uv.lock will be generally quite regularly > >>> bumped and "eventually consistent" with constraints. > >>> > >>> > >>> *Benefits:* > >>> * simplification of breeze build image with constraints (currently it > >> uses > >>> complex uv pip install with --constraint, and it could simply use uv > sync > >>> --all-packages instead if our uv.lock is checked in and maintained > >>> * better synchronization of the local venv. Currently uv sync does not > >>> guarantee to have the same version of dependencies in local venv. The > >> only > >>> way to guarantee it now is to use breeze image - build with constraints > >> or > >>> run complex commands with constraints added when you run uv sync > >>> * less "3rd-party release" impact on failing PRs. if PRs will not touch > >> the > >>> dependency specification where the new release is released, it will > >>> continue using the dependency stored in .lock and not the latest > >> dependency > >>> released > >>> > >>> > >>> *Cons:* > >>> * Probably we should pay more attention to run > upgrade-important-versions > >>> regularly and even add an option there to manually just perform the > sync > >>> (for example that will be useful as part of the release procedure, > where > >>> release manager would have to make sure to run the > >>> upgrade-important-versions check when preparing release notes. But it > has > >>> the advantage that it will be a conscious step that should actually > help > >> in > >>> making sure that constraints are updated. > >>> > >>> * sometimes PRs of people will have "big" changes to uv.lock - when > they > >>> are updating the dependency and alongside the uv.lock will also bump > many > >>> other unrelated dependencies - this could be mitigated by splitting it > >> into > >>> "lock upgrade" and "dependency update" but workflow of that is not very > >>> convenient and natural. > >>> > >>> J. > >> > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > >> For additional commands, e-mail: dev-h...@airflow.apache.org > >> > >> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > For additional commands, e-mail: dev-h...@airflow.apache.org > >
