> pip is not aiming and > will likely never be a "development/project management" tool and
I’m a bit surprised. I thought the dep group thing was for this. >> 3) to give our users a way to install released airflow in reproducible way > be able to use `uv` export to pylock.toml and replace our `--constraint` > installation. I’m thinking of supporting both. Or at least for a while, till `pylock.toml` becomes well-known (which might take a long long time…) They’re generated files, so might not be that hard to maintain. Best, Wei > On Aug 19, 2025, at 5:45 PM, Jarek Potiuk <ja...@potiuk.com> wrote: > >> +1 to this idea. I considered whether we should use `pylock.toml` > instead, but it might not address the issue we intend to resolve based on > my reading. > > Yeah. pylock.toml and https://peps.python.org/pep-0751/ (in the future) are > better replacement for point 3: > >> 3) to give our users a way to install released airflow in reproducible way > > This is the intention of pylock.toml - it should eventually allow our users > to install airflow in a reproducible way so basically would replace the > constraints of ours. For now support for it is experimental in PyPI and it > might change, also there is a general consensus that it does not fully > replace the ".lock" files feature for poetry, uv, pip-tools when it comes > to local development, workspaces, monorepos (which don't even have > standardisation yet). Also `uv` supports PEP751 pylock.toml as a (loosy - > it removes a number of metada kept in uv.lock specific for development > workflows) export format from its uv.lock. > > My view (and also a lot of the information I got from packaging people i > interact with) - that each of the "development/project management" tools > (like uv/poetry etc.) will keep their own ".lock" format - simply because > there is no possible way to encompass all the needs they have in "generic > way". There was about 100 os pages discussion when PEP 751 was proposed and > uv team took a very active part in it - but there seem to be a number of > points that are not really "generic". For example pip is not aiming and > will likely never be a "development/project management" tool and it does > not have a "workspace project" - it will always be really "end-user > installatiion tool" as it's primary function - and the result of this > different approach is what PEP 751 aims for - to allow reproducible > installation for end users. > > So what I see as a natural step when we adopt `uv.lock` as development > "source of truth" and our "development/project management tool" - we will > be able to use `uv` export to pylock.toml and replace our `--constraint` > installation. But while several tools (both pip and uv) already support > exporting to the pylock.toml, there is no way YET to install packages using > such lock to actually install packages. Pip for example has this issue open > for it https://github.com/pypa/pip/issues/13334 and it waits for someone > to volunteer and implement it. So for us it would make sense to switch for > 3) from constraints to pylock.toml - but only after `pip` and other way > people install airflow will be able to use it. Before that - there is no > benefit of us using it. This might even stay like that: `uv.lock` for > development and pylock.toml published by us for users who want to install > airflow in reproducible way. The `pylock.toml` has a number of benefits > comparing to current constraint approach - single lock supporting multiple > python versions, cryptographic integrity and attestation support, support > for multiple platforms, support for dependency groups, support for default > groups - all that will make few of our "custom" mechanisms we currently > "workaround" into the "solid features" of reproducible installation. But we > are not there yet :).\ > > J. > > > On Tue, Aug 19, 2025 at 10:59 AM Wei Lee <weilee...@gmail.com> wrote: > >> +1 to this idea. I considered whether we should use `pylock.toml` instead, >> but it might not address the issue we intend to resolve based on my reading. >> >> Best, >> Wei >> >>> On Aug 19, 2025, at 5:51 AM, Ferruzzi, Dennis >> <ferru...@amazon.com.INVALID> wrote: >>> >>> I don't have any particular experience, but based just on what you >> shared there, it does feel like a next logical step in the migration to >> these tools. Off-loading work and simplifying our CI is also always a plus >> in my opinion. >>> >>> >>> - ferruzzi >>> >>> >>> ________________________________ >>> From: Jarek Potiuk <ja...@potiuk.com> >>> Sent: Monday, August 18, 2025 4:14 AM >>> To: dev@airflow.apache.org >>> Subject: [EXT] [DISCUSS] Replace constraints with uv.lock mechanisms for >> dev env freeze >>> >>> CAUTION: This email originated from outside of the organization. Do not >> click links or open attachments unless you can confirm the sender and know >> the content is safe. >>> >>> >>> >>> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur >> externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous >> ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas >> certain que le contenu ne présente aucun risque. >>> >>> >>> >>> Hello here, >>> >>> *TL;DR: I would love to hear comments on the proposal to switch to using >>> uv.lock for our dev env dependency freeze* >>> >>> While working on some small follow-ups after prek upgrade, I think I >>> finally figured out how we can start using `uv.lock` locally and merge it >>> into our constraint mechanism effectively - without >>> over-burdening committers with merging a lot of "dependabot-like" PRs. >>> >>> The key problem that needed to be resolved was who and when would update >>> the uv.lock regularly. >>> >>> I captured my thoughts in https://github.com/apache/airflow/issues/54609 >>> >>> But let me copy the content of it below. I would love to hear the >> comments >>> - here or in the GitHub issue if you will (I will bring important things >>> from the issue here if they will be raised there): >>> >>> >>> *Currently constraints are used for three purposes:* >>> 1) to take snapshot of latest "working" version of dependencies >>> 2) to limit PRs to "latest known good constraints" - so that newly >> released >>> packages are not failing most PRs, but fail canary builds instead >>> 3) to give our users a way to install released airflow in reproducible >> way >>> >>> However, after we switched to uv, the case 2) can be better achieved with >>> uv.lock. Currently we .gitignore uv.lock, but we could commit it to the >>> repository and handle constraint generation differently >>> >>> * constraints could be generated using latest lock >>> * canary builds could use the --resolution highest as today and diverge >>> from latest "frozen" uv.lock >>> * all prs would use the uv.lock and do not use --resolution-highest in >>> those PRs. This has the added benefit that uv sync in PR would >>> automatically upgrade dependencies if the dependencies in pyproject.toml >>> are conflicting with uv.lock - but it would not use --resolution highest, >>> which means that the PRs that modify dependeNcies do not have to get >> latest >>> dependencies when they are run - that would limit the number of PRs that >>> are experiencing "main/canary" problems >>> >>> *Problem* >>> >>> The only problem to solve is who and when would upgrade the uv.lock and >>> commit it to the repo. Currently constraints are automatically bumped >> after >>> successful canary runs, but they are in orphan un-protected branches, >> while >>> uv.lock shares the code with main airflow repo, and we need a "human with >>> ICLA" to review and merge commits there (ASF requirement) >>> >>> >>> *Solution* >>> Proposed solution is to add such a uv.lock upgrade to our >>> "upgrade-important-versions" prek hook. We run it in CI in canary builds >>> and we fail the builds when new versions of important installers change >>> (pip, Python base images, golang) - this happens often-enough to be >>> eventually-consistent pretty fast, but rarely enough to make it into >>> annoying "everyday" upgrade chore. >>> >>> Regular PRs - especially those that would change dependencies will also >>> modify uv.lock - for example when a new dependency is added or removed or >>> bumped. This means that the uv.lock will be generally quite regularly >>> bumped and "eventually consistent" with constraints. >>> >>> >>> *Benefits:* >>> * simplification of breeze build image with constraints (currently it >> uses >>> complex uv pip install with --constraint, and it could simply use uv sync >>> --all-packages instead if our uv.lock is checked in and maintained >>> * better synchronization of the local venv. Currently uv sync does not >>> guarantee to have the same version of dependencies in local venv. The >> only >>> way to guarantee it now is to use breeze image - build with constraints >> or >>> run complex commands with constraints added when you run uv sync >>> * less "3rd-party release" impact on failing PRs. if PRs will not touch >> the >>> dependency specification where the new release is released, it will >>> continue using the dependency stored in .lock and not the latest >> dependency >>> released >>> >>> >>> *Cons:* >>> * Probably we should pay more attention to run upgrade-important-versions >>> regularly and even add an option there to manually just perform the sync >>> (for example that will be useful as part of the release procedure, where >>> release manager would have to make sure to run the >>> upgrade-important-versions check when preparing release notes. But it has >>> the advantage that it will be a conscious step that should actually help >> in >>> making sure that constraints are updated. >>> >>> * sometimes PRs of people will have "big" changes to uv.lock - when they >>> are updating the dependency and alongside the uv.lock will also bump many >>> other unrelated dependencies - this could be mitigated by splitting it >> into >>> "lock upgrade" and "dependency update" but workflow of that is not very >>> convenient and natural. >>> >>> J. >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org >> For additional commands, e-mail: dev-h...@airflow.apache.org >> >> --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org For additional commands, e-mail: dev-h...@airflow.apache.org
