> Quick question: We will still make the branches for constraints as we used to, correct?
Quick answer: Correct. Longer answer with more context and future vision. The constraints mechanism is currently used for both: 1) development (effectively pinning the dependencies for most PRs and only bumping them to latest in canary builds and in some PRs) 2) user facing reproducible installation with `pip` By - eventually - switching to uv we will be able to replace 1) with uv.lock, but still continue to generate the constraints as we did for 2). This means that we will "bind" development with `uv` but the "user" side of the installation of airflow will not require UV in any scenario, we are following "de-facto" standard of remote constraint mechanism (it's not really part of standard but something that both `pip` and `uv` support). Eventually we might be able to replace 2) for our users when it becomes standard and implemented by multiple tools like `pip`. This is essentially what PEP-751 is about: https://peps.python.org/pep-0751/ - Bret, TP and other packaging maintainers recognize our case as valid and they are designing the standard format that will be usable in both 1) and 2) cases. Our "constraint" mechanism is essentially a "poor man" version of PEP-751 implemented by me, in the absence of a similar mechanism in pip and packaging standards. This is about the last thing that the packaging team is catching up with all the needs Airflow had with packaging since Airflow 2 and providers introduction. This will be different than uv.lock, the current proposal is to use .toml format for it and - from what I know - uv team is involved and committed - you can see Charlie and Zanieb from uv very actively participating in those 3 super long threads: Original restart of PEP-751 (there was a previous PEP=665 https://peps.python.org/pep-0665/ that was rejected) * https://discuss.python.org/t/pep-751-lock-files-again/59173 (354 posts) Reboot: * https://discuss.python.org/t/pep-751-now-with-graphs/69721/254 (253 posts) And one last reboot : * https://discuss.python.org/t/pep-751-one-last-time/77293/133 (136 posts) I have not actively participated in those discussions - I was barely following them, and I think Charlie and Zanieb are already looking at Airflow as one of the most complex cases they will need to serve, so they are for sure doing a good job in the discussions led by Bret, Paul Moor and involving many other people, and I have a full trust the standard they will come with will be good for us :). You can see by the sheer number of posts how thorough and detailed the discussion was. So - once PEP-751 is approved and implemented by `pip`, `uv`, `poetry` and others, we will be able to replace the uv.lock mechanism with it and also use it for reproducible installation for our users. But this is again - a few years from now. J. On Sat, Mar 22, 2025 at 11:31 AM Abhishek Bhakat <abhishek.bha...@astronomer.io.invalid> wrote: > +1 for uv. > Quick question: We will still make the branches for constraints as we used > to, correct? > > - Avi > > On Sat, Mar 22, 2025 at 7:24 AM Jarek Potiuk <ja...@potiuk.com> wrote: > > > Also one more important point - if we drop `pip` support - we will be > able > > to rely on `uv.lock` for constraint generation. Currently we have our > own > > mechanism of updating the constraint files, but if we go "full-in" with > > `uv` - we should be able to use `uv.lock` for this purpose. It means that > > we will be able to commit `uv.lock` (currently it's .gitignored and > > everyone has its own uv.lock file). > > > > As of two weeks or so apparently dependabot supports uv > > > > > https://github.blog/changelog/2025-03-13-dependabot-version-updates-now-support-uv-in-general-availability/ > > though I > > am not sure if this is full support ( > > > https://docs.astral.sh/uv/guides/integration/dependency-bots/#uvlock-output > > mentions that it is work in progress and the issue > > https://github.com/dependabot/dependabot-core/issues/10478 mentioned > there > > is still open). > > > > Once we go "full-in" with `uv` and have full support for `uv.lock` > upgrade > > with dependabot, we will be able to replace the current "canary > > auto-upgrade" with "Dependabot PR upgrade" mechanism, which should allow > > us to remove a lot of custom code I wrote and will allow to have more > > "controlled" upgrades (say daily PR to upgrade deps). But details for > that > > is another discussion. > > > > J. > > > > > > On Sat, Mar 22, 2025 at 7:44 AM Kunal Bhattacharya < > kunal.ju...@gmail.com> > > wrote: > > > > > +1 on dropping pip support and have simplified dependency management > with > > > uv. > > > > > > I agree that managing both is complicated and the quick start docs to > > > navigate either uv or pip is not the easiest to follow through. I > > strongly > > > believe the setup will be orders of magnitude easier to follow along > with > > > just uv. > > > > > > Regards, > > > Kunal Bhattacharya > > > > > > On Sat, 22 Mar 2025, 03:20 Jarek Potiuk, <ja...@potiuk.com> wrote: > > > > > > > Hello here, > > > > > > > > Following the "airflow-core" move I would like to ask what others > think > > > > about dropping `pip` as a way to set-up your local dev environment > and > > > > leaving only `uv`. > > > > > > > > *Context: * > > > > > > > > The `uv` tool from https://astral.sh/ is built in ruff and the way > > they > > > > approached python dev environments is inspired by rust cargo. I.e > "one > > > tool > > > > to do it all". > > > > > > > > The uv tool does this: > > > > > > > > * automatically installs python when needed > > > > * automatically creates venv when needed > > > > * automatically syncs venv with your project dependencies when needed > > > > * supports workspaces - i.e. multiple python projects in a single > repo, > > > > which is something we started to use extensively recently and > > culminated > > > > with airflow-core separation > > > > > > > > This is all cool, and based mostly on implementing the standards > agreed > > > by > > > > the Python Packaging team (https://peps.python.org/topic/packaging/) > > - > > > > which means that we do not force our users and developers to use > > certain > > > > tools, as long as they follow standards. One can use pip, poetry, > > hatch, > > > > uv, flit and a number of others as a front-end client to install and > > > manage > > > > their local environment. > > > > > > > > This however breaks a bit with workspaces and dependency groups and > > lock > > > > files: > > > > > > > > * dependency groups (used for devel/ test dependencies) is already > > > approved > > > > https://peps.python.org/pep-0735/ but not implemented by `pip` yet > > (next > > > > version 25.1 will support it) > > > > > > > > * workspaces do not have a PEP proposal yet. Different tools > implement > > > them > > > > in different ways - I work with Ofek, Hatch creator and he has his > own > > > > ideas and implementations, currently uv's workspaces (modelled after > > the > > > > needs of ours) are working very well for us > > > > > > > > * uv.lock is an incarnation (by uv) of > > https://peps.python.org/pep-0751/ > > > > which is currently draft but there is a rumour it will be approved > soon > > > and > > > > uv team is committed to support the PEP > > > > > > > > *Problem:* > > > > > > > > Currently, a number of decisions in our build/packaging files are > > > > supporting the "generic" support. Everything we do (including > workspace > > > > installation) should be possible with `uv` > > > > but we also have `pip` equivalent of it (for example you have a > > sequence > > > of > > > > `pip install -e` commands that you can run to get the same result as > > `uv > > > > sync`). And it costs a bit - hatch_build.py in the root of Airflow is > > > > unnecessarily complex to support it (with dynamic pre-installed > > packages > > > > and some other dynamic code. We could get rid of it and replace the > > > dynamic > > > > stuff with static pyproject.toml in the root folder if we do not care > > > about > > > > `pip` installation for development. It would also simplify "quick > > start" > > > > docs if we only support `uv` with workspaces. > > > > > > > > The problem with `uv` is that it's an open-source, but privately > > > controlled > > > > (astral.sh) by a VC-backed company. We know (personally) and like the > > > > people behind `uv`, but at some future point in time, the ownership > and > > > > change of control might turn astral into > not-that-open-source-friendly > > > > (this is in a stark contrast with `pip`, `hatch`, `flit` - which are > > part > > > > of the Python Software Foundation and under the Python Packaging > > > Authority > > > > working group. > > > > > > > > *Concerns* > > > > > > > > I personally have some reservations against exclusively supporting > > `uv`, > > > > but I recognise that it makes our packaging unnecessarily complex if > we > > > > continue to support other workflows. It's great DevEX, fantastic > > > > "Contributor Journey Optimisation" with `uv`. > > > > > > > > The risks, however, are small. If anything happens with `astral` - > they > > > > licence it with Apache 2 and MIT dual licensing, we - or anyone else > - > > > > could fork it as it happened with multiple other projects in the past > > > > including Akka -> Pekko, Terraform -> OpenTofu and numerous others. > > > > > > > > And it does not affect our users. Packages published from us are > > > > installable with anything (including pip, hatch, flit) because we > > follow > > > > standards, it's only about the dev tooling. And - if all things fail, > > we > > > > can always redevelop our own "glue" (like we already do in breeze) to > > > allow > > > > workspace or dependency groups to be easily usable. > > > > > > > > *Proposal* > > > > > > > > My proposal - for the sake of simplicity, better contributor's > journey > > - > > > > for now to drop `pip` support and have `uv` as the only supported dev > > > > tooling. That would allow us to simplify docs and tooling support. > > > > > > > > I would love to hear what others think about it. > > > > > > > > J. > > > > > > > > > >
