Yeah. Dependabot has support for it and I am already testing it for a while with breeze's `uv.lock` - so - to add a bit of context - I started slow by commitiung `uv.lock` in breeze and seeing the kind of issues we might have. One of the issues there is that I am not entirely sure if we want dependabot to upgrade things for us for the whole "workspace", I find this auto-upgrade that is happening currently way better - because it literally gets us few hours feedback when things are broken by packages released by 3rd-parties - and we know when they break our tests, otherwise it's pretty transparent.
If you look at our constraints history https://github.com/apache/airflow/commits/constraints-main/ we have 3-4 constraints upgrades a DAY(!) and many of those upgrades are more than one dependency. And this is really nice because those are literally "checkpoints" of "dependencies we knew passed all the tests". So when someone releases a dependency that breaks something we do not need to look far - our "canary" build shows exactly what's changed since the last "good canary run" - which means that we can identify and fix the problems super quickly - because we know that this particular dependency upgrade caused it. And it's super easy to test with breeze - you just reproduce it failing, downgrade the dep and see it working - and you know 100% that's it. This follows one of my favourite saying - "if something is painful, do it more often" - this is why resurrecting and old branch (I am dreading a bit the moment I will start preparing 2.11.1) is often quite painful - because you see many things broken and you have to track it through 100s of dependencies upgraded. Current way Dependabot works will not give us the same experience though - it's either going to be very annoying (with dependabot PR upgrade every single day at the very least) - or causing longer investigations (if we batch it for - say - weekly upgrades). None of those two is even close to the current experience with constraints where things **just work** - until they don't and then you know what to do. So what's I am aiming at in this "uv.lock" change is not even to use dependabot for it, but to fulfill those two seemingly contradictory goals: * keep the seamless experience of upgrades in "canary" and allowing us to diagnose and fix failures quickly * but give even more stability and "standard" experience with `uv sync` and `uv lock` features so that contributors and committers can get more "stable" and more "works for me syndrom free" experience. And I think what I proposed will allow us to eat cake and have it too - what I proposed (without even having dependabot involved) will actually do the job quite nicely on both fronts. BTW. We will still likely use dependabot for airflow deps but **only** to address security needs I think. We are already on a streak to squash and remove all kinds of blockers to upgrade things, but having dependabot flag it for us automatically would be quite nice. J. On Tue, Aug 19, 2025 at 2:48 PM Pavankumar Gopidesu <gopidesupa...@gmail.com> wrote: > Thanks Jarek for bringing this up, I really like this idea we should use > uv.lock. > > For upgrading packages in the uv.lock file, dependaboat started adding > support for this, but it seems there are quite a lot of issues. > Some i have seen dependaboat updates lock file but not updating the package > in toml file etc; there may be other issues similar to related. but I hope > these may be addressed at some point of time ( Not sure when :) ) > > https://docs.astral.sh/uv/guides/integration/dependency-bots/#dependabot > https://github.com/astral-sh/uv/issues/2512 > > https://github.com/dependabot/dependabot-core/issues?q=is%3Aissue%20state%3Aopen%20uv > > Pavan > > On Tue, Aug 19, 2025 at 11:18 AM Wei Lee <weilee...@gmail.com> wrote: > > > > pip is not aiming and > > > will likely never be a "development/project management" tool and > > > > I’m a bit surprised. I thought the dep group thing was for this. > > > > >> 3) to give our users a way to install released airflow in reproducible > > way > > > > > be able to use `uv` export to pylock.toml and replace our > `--constraint` > > > installation. > > > > I’m thinking of supporting both. Or at least for a while, till > > `pylock.toml` becomes well-known (which might take a long long time…) > > They’re generated files, so might not be that hard to maintain. > > > > > > Best, > > Wei > > > > > On Aug 19, 2025, at 5:45 PM, Jarek Potiuk <ja...@potiuk.com> wrote: > > > > > >> +1 to this idea. I considered whether we should use `pylock.toml` > > > instead, but it might not address the issue we intend to resolve based > on > > > my reading. > > > > > > Yeah. pylock.toml and https://peps.python.org/pep-0751/ (in the > future) > > are > > > better replacement for point 3: > > > > > >> 3) to give our users a way to install released airflow in reproducible > > way > > > > > > This is the intention of pylock.toml - it should eventually allow our > > users > > > to install airflow in a reproducible way so basically would replace the > > > constraints of ours. For now support for it is experimental in PyPI and > > it > > > might change, also there is a general consensus that it does not fully > > > replace the ".lock" files feature for poetry, uv, pip-tools when it > comes > > > to local development, workspaces, monorepos (which don't even have > > > standardisation yet). Also `uv` supports PEP751 pylock.toml as a > (loosy - > > > it removes a number of metada kept in uv.lock specific for development > > > workflows) export format from its uv.lock. > > > > > > My view (and also a lot of the information I got from packaging people > i > > > interact with) - that each of the "development/project management" > tools > > > (like uv/poetry etc.) will keep their own ".lock" format - simply > because > > > there is no possible way to encompass all the needs they have in > "generic > > > way". There was about 100 os pages discussion when PEP 751 was proposed > > and > > > uv team took a very active part in it - but there seem to be a number > of > > > points that are not really "generic". For example pip is not aiming and > > > will likely never be a "development/project management" tool and it > does > > > not have a "workspace project" - it will always be really "end-user > > > installatiion tool" as it's primary function - and the result of this > > > different approach is what PEP 751 aims for - to allow reproducible > > > installation for end users. > > > > > > So what I see as a natural step when we adopt `uv.lock` as development > > > "source of truth" and our "development/project management tool" - we > will > > > be able to use `uv` export to pylock.toml and replace our > `--constraint` > > > installation. But while several tools (both pip and uv) already support > > > exporting to the pylock.toml, there is no way YET to install packages > > using > > > such lock to actually install packages. Pip for example has this issue > > open > > > for it https://github.com/pypa/pip/issues/13334 and it waits for > > someone > > > to volunteer and implement it. So for us it would make sense to switch > > for > > > 3) from constraints to pylock.toml - but only after `pip` and other way > > > people install airflow will be able to use it. Before that - there is > no > > > benefit of us using it. This might even stay like that: `uv.lock` for > > > development and pylock.toml published by us for users who want to > install > > > airflow in reproducible way. The `pylock.toml` has a number of benefits > > > comparing to current constraint approach - single lock supporting > > multiple > > > python versions, cryptographic integrity and attestation support, > support > > > for multiple platforms, support for dependency groups, support for > > default > > > groups - all that will make few of our "custom" mechanisms we > currently > > > "workaround" into the "solid features" of reproducible installation. > But > > we > > > are not there yet :).\ > > > > > > J. > > > > > > > > > On Tue, Aug 19, 2025 at 10:59 AM Wei Lee <weilee...@gmail.com> wrote: > > > > > >> +1 to this idea. I considered whether we should use `pylock.toml` > > instead, > > >> but it might not address the issue we intend to resolve based on my > > reading. > > >> > > >> Best, > > >> Wei > > >> > > >>> On Aug 19, 2025, at 5:51 AM, Ferruzzi, Dennis > > >> <ferru...@amazon.com.INVALID> wrote: > > >>> > > >>> I don't have any particular experience, but based just on what you > > >> shared there, it does feel like a next logical step in the migration > to > > >> these tools. Off-loading work and simplifying our CI is also always a > > plus > > >> in my opinion. > > >>> > > >>> > > >>> - ferruzzi > > >>> > > >>> > > >>> ________________________________ > > >>> From: Jarek Potiuk <ja...@potiuk.com> > > >>> Sent: Monday, August 18, 2025 4:14 AM > > >>> To: dev@airflow.apache.org > > >>> Subject: [EXT] [DISCUSS] Replace constraints with uv.lock mechanisms > > for > > >> dev env freeze > > >>> > > >>> CAUTION: This email originated from outside of the organization. Do > not > > >> click links or open attachments unless you can confirm the sender and > > know > > >> the content is safe. > > >>> > > >>> > > >>> > > >>> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur > > >> externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si > > vous > > >> ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes > pas > > >> certain que le contenu ne présente aucun risque. > > >>> > > >>> > > >>> > > >>> Hello here, > > >>> > > >>> *TL;DR: I would love to hear comments on the proposal to switch to > > using > > >>> uv.lock for our dev env dependency freeze* > > >>> > > >>> While working on some small follow-ups after prek upgrade, I think I > > >>> finally figured out how we can start using `uv.lock` locally and > merge > > it > > >>> into our constraint mechanism effectively - without > > >>> over-burdening committers with merging a lot of "dependabot-like" > PRs. > > >>> > > >>> The key problem that needed to be resolved was who and when would > > update > > >>> the uv.lock regularly. > > >>> > > >>> I captured my thoughts in > > https://github.com/apache/airflow/issues/54609 > > >>> > > >>> But let me copy the content of it below. I would love to hear the > > >> comments > > >>> - here or in the GitHub issue if you will (I will bring important > > things > > >>> from the issue here if they will be raised there): > > >>> > > >>> > > >>> *Currently constraints are used for three purposes:* > > >>> 1) to take snapshot of latest "working" version of dependencies > > >>> 2) to limit PRs to "latest known good constraints" - so that newly > > >> released > > >>> packages are not failing most PRs, but fail canary builds instead > > >>> 3) to give our users a way to install released airflow in > reproducible > > >> way > > >>> > > >>> However, after we switched to uv, the case 2) can be better achieved > > with > > >>> uv.lock. Currently we .gitignore uv.lock, but we could commit it to > the > > >>> repository and handle constraint generation differently > > >>> > > >>> * constraints could be generated using latest lock > > >>> * canary builds could use the --resolution highest as today and > diverge > > >>> from latest "frozen" uv.lock > > >>> * all prs would use the uv.lock and do not use --resolution-highest > in > > >>> those PRs. This has the added benefit that uv sync in PR would > > >>> automatically upgrade dependencies if the dependencies in > > pyproject.toml > > >>> are conflicting with uv.lock - but it would not use --resolution > > highest, > > >>> which means that the PRs that modify dependeNcies do not have to get > > >> latest > > >>> dependencies when they are run - that would limit the number of PRs > > that > > >>> are experiencing "main/canary" problems > > >>> > > >>> *Problem* > > >>> > > >>> The only problem to solve is who and when would upgrade the uv.lock > and > > >>> commit it to the repo. Currently constraints are automatically bumped > > >> after > > >>> successful canary runs, but they are in orphan un-protected branches, > > >> while > > >>> uv.lock shares the code with main airflow repo, and we need a "human > > with > > >>> ICLA" to review and merge commits there (ASF requirement) > > >>> > > >>> > > >>> *Solution* > > >>> Proposed solution is to add such a uv.lock upgrade to our > > >>> "upgrade-important-versions" prek hook. We run it in CI in canary > > builds > > >>> and we fail the builds when new versions of important installers > change > > >>> (pip, Python base images, golang) - this happens often-enough to be > > >>> eventually-consistent pretty fast, but rarely enough to make it into > > >>> annoying "everyday" upgrade chore. > > >>> > > >>> Regular PRs - especially those that would change dependencies will > also > > >>> modify uv.lock - for example when a new dependency is added or > removed > > or > > >>> bumped. This means that the uv.lock will be generally quite regularly > > >>> bumped and "eventually consistent" with constraints. > > >>> > > >>> > > >>> *Benefits:* > > >>> * simplification of breeze build image with constraints (currently it > > >> uses > > >>> complex uv pip install with --constraint, and it could simply use uv > > sync > > >>> --all-packages instead if our uv.lock is checked in and maintained > > >>> * better synchronization of the local venv. Currently uv sync does > not > > >>> guarantee to have the same version of dependencies in local venv. The > > >> only > > >>> way to guarantee it now is to use breeze image - build with > constraints > > >> or > > >>> run complex commands with constraints added when you run uv sync > > >>> * less "3rd-party release" impact on failing PRs. if PRs will not > touch > > >> the > > >>> dependency specification where the new release is released, it will > > >>> continue using the dependency stored in .lock and not the latest > > >> dependency > > >>> released > > >>> > > >>> > > >>> *Cons:* > > >>> * Probably we should pay more attention to run > > upgrade-important-versions > > >>> regularly and even add an option there to manually just perform the > > sync > > >>> (for example that will be useful as part of the release procedure, > > where > > >>> release manager would have to make sure to run the > > >>> upgrade-important-versions check when preparing release notes. But it > > has > > >>> the advantage that it will be a conscious step that should actually > > help > > >> in > > >>> making sure that constraints are updated. > > >>> > > >>> * sometimes PRs of people will have "big" changes to uv.lock - when > > they > > >>> are updating the dependency and alongside the uv.lock will also bump > > many > > >>> other unrelated dependencies - this could be mitigated by splitting > it > > >> into > > >>> "lock upgrade" and "dependency update" but workflow of that is not > very > > >>> convenient and natural. > > >>> > > >>> J. > > >> > > >> > > >> --------------------------------------------------------------------- > > >> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > >> For additional commands, e-mail: dev-h...@airflow.apache.org > > >> > > >> > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > For additional commands, e-mail: dev-h...@airflow.apache.org > > > > >
