Thanks Jarek, that all makes sense :) Looking forward to it. happy to help where I can.
Pavan On Tue, Aug 19, 2025 at 9:56 PM Jarek Potiuk <ja...@potiuk.com> wrote: > > > > > > > > (1) If we plan to split the distributions to separate api-server, > > scheduler deployments in future - will we then also as next increment > > the split into multiple logical uv.locks? Or will it be (for the time > > being) keep a gloabal uv-lock for all and everything in the repo? > > > > This should be for sure one workspace and one uv.lock. The main > reason (and the best feature of workspace) is that it puts all the > distributions we have and figure out a set of dependencies that are > not conflicting for ALL of them. This means that we will never get > into the situation that we cannot install them all together - and despite > of splitting the distributions we still want to be able to do > > pip install apache-airflow[all] > > And have everything installed in CI image. > > However, we might indeed decide to keep several uv.lock files (for > example breeze has its own uv.lock file and it **works** - but it is > mostly for testing). We could also have a separate one for airflow-ctl > But there is a limited use of that (not unless we have pylock.toml) > to be honest and it is nice to have a single CI image where > everything is installed in one env and we can run it all in the image. > > > (2) I have a bit of worries if somebody in a PR adds a dependency and > > the uv-lock is a brader upgrade - we need to ensure if uv-lock is > > touched such PR needs to run all tests in all versions? > > > > We can. We already do that in fact - currently when you modify any > of the pyproject.toml files, selective checks already set the > "upgrade-to-newer-dependencies" output, and it triggers all tests, all > versions > and we can update it to be a bit more selective and work out right > selective checks here. > > > > (3) is the uv.lock Python version dependent? Because we need to have a > > lock file per Python version. > > > > It contains all combinations of versions and architectures and potentially > other parameters that can make different resolution works. > > > > > > Other than this I hope that the change will be one that slims complexity > > of dev tooling codebase with a minimal risk that we run into a uv-vendor > > lock... but the way back is not blocked will just be effort if astral > > turns into evil in the future. > > > > Yeah. That is a valid worry. It only touches the development side for now > and this makes it easy to revert. Honestly it does not lock us "too much". > Constraints are generally generated from 'installed" packages (we do > essentially `pip freeze` in ci-image (simplified - it's a bit more complex > than that and we have 3 variants of those). Currently we already use > a very long `uv pip install a b c d --constraint` (list of packages derived > from > list of pyproject.toml we have to get that environment and we could > do it with `pip` (way slower but pip is implementing some of the speedups > that uv already did). This change is going to replace it with - essentially > `uv sync --frozen --all-packages` . And we have history of those > changes so we can always go back. It would mean + few 100 > lines of code or so, but nothing major. > > > > > > Jens > > > > On 19.08.25 16:12, Jarek Potiuk wrote: > > > Yeah. Dependabot has support for it and I am already testing it for a > > while > > > with breeze's `uv.lock` - so - to add a bit of context - I started slow > > by > > > commitiung `uv.lock` in breeze and seeing the kind of issues we might > > have. > > > One of the issues there is that I am not entirely sure if we want > > > dependabot to upgrade things for us for the whole "workspace", I find > > this > > > auto-upgrade that is happening currently way better - because it > > literally > > > gets us few hours feedback when things are broken by packages released > by > > > 3rd-parties - and we know when they break our tests, otherwise it's > > pretty > > > transparent. > > > > > > If you look at our constraints history > > > https://github.com/apache/airflow/commits/constraints-main/ we have > 3-4 > > > constraints upgrades a DAY(!) and many of those upgrades are more than > > one > > > dependency. And this is really nice because those are literally > > > "checkpoints" of "dependencies we knew passed all the tests". So when > > > someone releases a dependency that breaks something we do not need to > > look > > > far - our "canary" build shows exactly what's changed since the last > > "good > > > canary run" - which means that we can identify and fix the problems > super > > > quickly - because we know that this particular dependency upgrade > caused > > > it. And it's super easy to test with breeze - you just reproduce it > > > failing, downgrade the dep and see it working - and you know 100% > that's > > it. > > > > > > This follows one of my favourite saying - "if something is painful, do > it > > > more often" - this is why resurrecting and old branch (I am dreading a > > bit > > > the moment I will start preparing 2.11.1) is often quite painful - > > because > > > you see many things broken and you have to track it through 100s of > > > dependencies upgraded. > > > > > > Current way Dependabot works will not give us the same experience > though > > - > > > it's either going to be very annoying (with dependabot PR upgrade every > > > single day at the very least) - or causing longer investigations (if we > > > batch it for - say - weekly upgrades). None of those two is even close > to > > > the current experience with constraints where things **just work** - > > until > > > they don't and then you know what to do. > > > > > > So what's I am aiming at in this "uv.lock" change is not even to use > > > dependabot for it, but to fulfill those two seemingly contradictory > > goals: > > > > > > * keep the seamless experience of upgrades in "canary" and allowing us > to > > > diagnose and fix failures quickly > > > * but give even more stability and "standard" experience with `uv sync` > > and > > > `uv lock` features so that contributors and committers can get more > > > "stable" and more "works for me syndrom free" experience. > > > > > > And I think what I proposed will allow us to eat cake and have it too - > > > what I proposed (without even having dependabot involved) will actually > > do > > > the job quite nicely on both fronts. > > > > > > BTW. We will still likely use dependabot for airflow deps but **only** > to > > > address security needs I think. We are already on a streak to squash > and > > > remove all kinds of blockers to upgrade things, but having dependabot > > flag > > > it for us automatically would be quite nice. > > > > > > J. > > > > > > > > > > > > On Tue, Aug 19, 2025 at 2:48 PM Pavankumar Gopidesu < > > gopidesupa...@gmail.com> > > > wrote: > > > > > >> Thanks Jarek for bringing this up, I really like this idea we should > use > > >> uv.lock. > > >> > > >> For upgrading packages in the uv.lock file, dependaboat started adding > > >> support for this, but it seems there are quite a lot of issues. > > >> Some i have seen dependaboat updates lock file but not updating the > > package > > >> in toml file etc; there may be other issues similar to related. but I > > hope > > >> these may be addressed at some point of time ( Not sure when :) ) > > >> > > >> > > https://docs.astral.sh/uv/guides/integration/dependency-bots/#dependabot > > >> https://github.com/astral-sh/uv/issues/2512 > > >> > > >> > > > https://github.com/dependabot/dependabot-core/issues?q=is%3Aissue%20state%3Aopen%20uv > > >> > > >> Pavan > > >> > > >> On Tue, Aug 19, 2025 at 11:18 AM Wei Lee <weilee...@gmail.com> wrote: > > >> > > >>>> pip is not aiming and > > >>>> will likely never be a "development/project management" tool and > > >>> I’m a bit surprised. I thought the dep group thing was for this. > > >>> > > >>>>> 3) to give our users a way to install released airflow in > > reproducible > > >>> way > > >>> > > >>>> be able to use `uv` export to pylock.toml and replace our > > >> `--constraint` > > >>>> installation. > > >>> I’m thinking of supporting both. Or at least for a while, till > > >>> `pylock.toml` becomes well-known (which might take a long long time…) > > >>> They’re generated files, so might not be that hard to maintain. > > >>> > > >>> > > >>> Best, > > >>> Wei > > >>> > > >>>> On Aug 19, 2025, at 5:45 PM, Jarek Potiuk <ja...@potiuk.com> wrote: > > >>>> > > >>>>> +1 to this idea. I considered whether we should use `pylock.toml` > > >>>> instead, but it might not address the issue we intend to resolve > based > > >> on > > >>>> my reading. > > >>>> > > >>>> Yeah. pylock.toml and https://peps.python.org/pep-0751/ (in the > > >> future) > > >>> are > > >>>> better replacement for point 3: > > >>>> > > >>>>> 3) to give our users a way to install released airflow in > > reproducible > > >>> way > > >>>> This is the intention of pylock.toml - it should eventually allow > our > > >>> users > > >>>> to install airflow in a reproducible way so basically would replace > > the > > >>>> constraints of ours. For now support for it is experimental in PyPI > > and > > >>> it > > >>>> might change, also there is a general consensus that it does not > fully > > >>>> replace the ".lock" files feature for poetry, uv, pip-tools when it > > >> comes > > >>>> to local development, workspaces, monorepos (which don't even have > > >>>> standardisation yet). Also `uv` supports PEP751 pylock.toml as a > > >> (loosy - > > >>>> it removes a number of metada kept in uv.lock specific for > development > > >>>> workflows) export format from its uv.lock. > > >>>> > > >>>> My view (and also a lot of the information I got from packaging > people > > >> i > > >>>> interact with) - that each of the "development/project management" > > >> tools > > >>>> (like uv/poetry etc.) will keep their own ".lock" format - simply > > >> because > > >>>> there is no possible way to encompass all the needs they have in > > >> "generic > > >>>> way". There was about 100 os pages discussion when PEP 751 was > > proposed > > >>> and > > >>>> uv team took a very active part in it - but there seem to be a > number > > >> of > > >>>> points that are not really "generic". For example pip is not aiming > > and > > >>>> will likely never be a "development/project management" tool and it > > >> does > > >>>> not have a "workspace project" - it will always be really "end-user > > >>>> installatiion tool" as it's primary function - and the result of > this > > >>>> different approach is what PEP 751 aims for - to allow reproducible > > >>>> installation for end users. > > >>>> > > >>>> So what I see as a natural step when we adopt `uv.lock` as > development > > >>>> "source of truth" and our "development/project management tool" - we > > >> will > > >>>> be able to use `uv` export to pylock.toml and replace our > > >> `--constraint` > > >>>> installation. But while several tools (both pip and uv) already > > support > > >>>> exporting to the pylock.toml, there is no way YET to install > packages > > >>> using > > >>>> such lock to actually install packages. Pip for example has this > issue > > >>> open > > >>>> for it https://github.com/pypa/pip/issues/13334 and it waits for > > >>> someone > > >>>> to volunteer and implement it. So for us it would make sense to > switch > > >>> for > > >>>> 3) from constraints to pylock.toml - but only after `pip` and other > > way > > >>>> people install airflow will be able to use it. Before that - there > is > > >> no > > >>>> benefit of us using it. This might even stay like that: `uv.lock` > for > > >>>> development and pylock.toml published by us for users who want to > > >> install > > >>>> airflow in reproducible way. The `pylock.toml` has a number of > > benefits > > >>>> comparing to current constraint approach - single lock supporting > > >>> multiple > > >>>> python versions, cryptographic integrity and attestation support, > > >> support > > >>>> for multiple platforms, support for dependency groups, support for > > >>> default > > >>>> groups - all that will make few of our "custom" mechanisms we > > >> currently > > >>>> "workaround" into the "solid features" of reproducible installation. > > >> But > > >>> we > > >>>> are not there yet :).\ > > >>>> > > >>>> J. > > >>>> > > >>>> > > >>>> On Tue, Aug 19, 2025 at 10:59 AM Wei Lee <weilee...@gmail.com> > wrote: > > >>>> > > >>>>> +1 to this idea. I considered whether we should use `pylock.toml` > > >>> instead, > > >>>>> but it might not address the issue we intend to resolve based on my > > >>> reading. > > >>>>> Best, > > >>>>> Wei > > >>>>> > > >>>>>> On Aug 19, 2025, at 5:51 AM, Ferruzzi, Dennis > > >>>>> <ferru...@amazon.com.INVALID> wrote: > > >>>>>> I don't have any particular experience, but based just on what you > > >>>>> shared there, it does feel like a next logical step in the > migration > > >> to > > >>>>> these tools. Off-loading work and simplifying our CI is also > always > > a > > >>> plus > > >>>>> in my opinion. > > >>>>>> > > >>>>>> - ferruzzi > > >>>>>> > > >>>>>> > > >>>>>> ________________________________ > > >>>>>> From: Jarek Potiuk <ja...@potiuk.com> > > >>>>>> Sent: Monday, August 18, 2025 4:14 AM > > >>>>>> To: dev@airflow.apache.org > > >>>>>> Subject: [EXT] [DISCUSS] Replace constraints with uv.lock > mechanisms > > >>> for > > >>>>> dev env freeze > > >>>>>> CAUTION: This email originated from outside of the organization. > Do > > >> not > > >>>>> click links or open attachments unless you can confirm the sender > and > > >>> know > > >>>>> the content is safe. > > >>>>>> > > >>>>>> > > >>>>>> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur > > >>>>> externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe > si > > >>> vous > > >>>>> ne pouvez pas confirmer l’identité de l’expéditeur et si vous > n’êtes > > >> pas > > >>>>> certain que le contenu ne présente aucun risque. > > >>>>>> > > >>>>>> > > >>>>>> Hello here, > > >>>>>> > > >>>>>> *TL;DR: I would love to hear comments on the proposal to switch to > > >>> using > > >>>>>> uv.lock for our dev env dependency freeze* > > >>>>>> > > >>>>>> While working on some small follow-ups after prek upgrade, I > think I > > >>>>>> finally figured out how we can start using `uv.lock` locally and > > >> merge > > >>> it > > >>>>>> into our constraint mechanism effectively - without > > >>>>>> over-burdening committers with merging a lot of "dependabot-like" > > >> PRs. > > >>>>>> The key problem that needed to be resolved was who and when would > > >>> update > > >>>>>> the uv.lock regularly. > > >>>>>> > > >>>>>> I captured my thoughts in > > >>> https://github.com/apache/airflow/issues/54609 > > >>>>>> But let me copy the content of it below. I would love to hear the > > >>>>> comments > > >>>>>> - here or in the GitHub issue if you will (I will bring important > > >>> things > > >>>>>> from the issue here if they will be raised there): > > >>>>>> > > >>>>>> > > >>>>>> *Currently constraints are used for three purposes:* > > >>>>>> 1) to take snapshot of latest "working" version of dependencies > > >>>>>> 2) to limit PRs to "latest known good constraints" - so that newly > > >>>>> released > > >>>>>> packages are not failing most PRs, but fail canary builds instead > > >>>>>> 3) to give our users a way to install released airflow in > > >> reproducible > > >>>>> way > > >>>>>> However, after we switched to uv, the case 2) can be better > achieved > > >>> with > > >>>>>> uv.lock. Currently we .gitignore uv.lock, but we could commit it > to > > >> the > > >>>>>> repository and handle constraint generation differently > > >>>>>> > > >>>>>> * constraints could be generated using latest lock > > >>>>>> * canary builds could use the --resolution highest as today and > > >> diverge > > >>>>>> from latest "frozen" uv.lock > > >>>>>> * all prs would use the uv.lock and do not use > --resolution-highest > > >> in > > >>>>>> those PRs. This has the added benefit that uv sync in PR would > > >>>>>> automatically upgrade dependencies if the dependencies in > > >>> pyproject.toml > > >>>>>> are conflicting with uv.lock - but it would not use --resolution > > >>> highest, > > >>>>>> which means that the PRs that modify dependeNcies do not have to > get > > >>>>> latest > > >>>>>> dependencies when they are run - that would limit the number of > PRs > > >>> that > > >>>>>> are experiencing "main/canary" problems > > >>>>>> > > >>>>>> *Problem* > > >>>>>> > > >>>>>> The only problem to solve is who and when would upgrade the > uv.lock > > >> and > > >>>>>> commit it to the repo. Currently constraints are automatically > > bumped > > >>>>> after > > >>>>>> successful canary runs, but they are in orphan un-protected > > branches, > > >>>>> while > > >>>>>> uv.lock shares the code with main airflow repo, and we need a > "human > > >>> with > > >>>>>> ICLA" to review and merge commits there (ASF requirement) > > >>>>>> > > >>>>>> > > >>>>>> *Solution* > > >>>>>> Proposed solution is to add such a uv.lock upgrade to our > > >>>>>> "upgrade-important-versions" prek hook. We run it in CI in canary > > >>> builds > > >>>>>> and we fail the builds when new versions of important installers > > >> change > > >>>>>> (pip, Python base images, golang) - this happens often-enough to > be > > >>>>>> eventually-consistent pretty fast, but rarely enough to make it > into > > >>>>>> annoying "everyday" upgrade chore. > > >>>>>> > > >>>>>> Regular PRs - especially those that would change dependencies will > > >> also > > >>>>>> modify uv.lock - for example when a new dependency is added or > > >> removed > > >>> or > > >>>>>> bumped. This means that the uv.lock will be generally quite > > regularly > > >>>>>> bumped and "eventually consistent" with constraints. > > >>>>>> > > >>>>>> > > >>>>>> *Benefits:* > > >>>>>> * simplification of breeze build image with constraints (currently > > it > > >>>>> uses > > >>>>>> complex uv pip install with --constraint, and it could simply use > uv > > >>> sync > > >>>>>> --all-packages instead if our uv.lock is checked in and maintained > > >>>>>> * better synchronization of the local venv. Currently uv sync does > > >> not > > >>>>>> guarantee to have the same version of dependencies in local venv. > > The > > >>>>> only > > >>>>>> way to guarantee it now is to use breeze image - build with > > >> constraints > > >>>>> or > > >>>>>> run complex commands with constraints added when you run uv sync > > >>>>>> * less "3rd-party release" impact on failing PRs. if PRs will not > > >> touch > > >>>>> the > > >>>>>> dependency specification where the new release is released, it > will > > >>>>>> continue using the dependency stored in .lock and not the latest > > >>>>> dependency > > >>>>>> released > > >>>>>> > > >>>>>> > > >>>>>> *Cons:* > > >>>>>> * Probably we should pay more attention to run > > >>> upgrade-important-versions > > >>>>>> regularly and even add an option there to manually just perform > the > > >>> sync > > >>>>>> (for example that will be useful as part of the release procedure, > > >>> where > > >>>>>> release manager would have to make sure to run the > > >>>>>> upgrade-important-versions check when preparing release notes. But > > it > > >>> has > > >>>>>> the advantage that it will be a conscious step that should > actually > > >>> help > > >>>>> in > > >>>>>> making sure that constraints are updated. > > >>>>>> > > >>>>>> * sometimes PRs of people will have "big" changes to uv.lock - > when > > >>> they > > >>>>>> are updating the dependency and alongside the uv.lock will also > bump > > >>> many > > >>>>>> other unrelated dependencies - this could be mitigated by > splitting > > >> it > > >>>>> into > > >>>>>> "lock upgrade" and "dependency update" but workflow of that is not > > >> very > > >>>>>> convenient and natural. > > >>>>>> > > >>>>>> J. > > >>>>> > > >>>>> > --------------------------------------------------------------------- > > >>>>> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > >>>>> For additional commands, e-mail: dev-h...@airflow.apache.org > > >>>>> > > >>>>> > > >>> > > >>> --------------------------------------------------------------------- > > >>> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > >>> For additional commands, e-mail: dev-h...@airflow.apache.org > > >>> > > >>> > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > For additional commands, e-mail: dev-h...@airflow.apache.org > > > > >
