+1 on dropping pip support and have simplified dependency management with uv.
I agree that managing both is complicated and the quick start docs to navigate either uv or pip is not the easiest to follow through. I strongly believe the setup will be orders of magnitude easier to follow along with just uv. Regards, Kunal Bhattacharya On Sat, 22 Mar 2025, 03:20 Jarek Potiuk, <ja...@potiuk.com> wrote: > Hello here, > > Following the "airflow-core" move I would like to ask what others think > about dropping `pip` as a way to set-up your local dev environment and > leaving only `uv`. > > *Context: * > > The `uv` tool from https://astral.sh/ is built in ruff and the way they > approached python dev environments is inspired by rust cargo. I.e "one tool > to do it all". > > The uv tool does this: > > * automatically installs python when needed > * automatically creates venv when needed > * automatically syncs venv with your project dependencies when needed > * supports workspaces - i.e. multiple python projects in a single repo, > which is something we started to use extensively recently and culminated > with airflow-core separation > > This is all cool, and based mostly on implementing the standards agreed by > the Python Packaging team (https://peps.python.org/topic/packaging/) - > which means that we do not force our users and developers to use certain > tools, as long as they follow standards. One can use pip, poetry, hatch, > uv, flit and a number of others as a front-end client to install and manage > their local environment. > > This however breaks a bit with workspaces and dependency groups and lock > files: > > * dependency groups (used for devel/ test dependencies) is already approved > https://peps.python.org/pep-0735/ but not implemented by `pip` yet (next > version 25.1 will support it) > > * workspaces do not have a PEP proposal yet. Different tools implement them > in different ways - I work with Ofek, Hatch creator and he has his own > ideas and implementations, currently uv's workspaces (modelled after the > needs of ours) are working very well for us > > * uv.lock is an incarnation (by uv) of https://peps.python.org/pep-0751/ > which is currently draft but there is a rumour it will be approved soon and > uv team is committed to support the PEP > > *Problem:* > > Currently, a number of decisions in our build/packaging files are > supporting the "generic" support. Everything we do (including workspace > installation) should be possible with `uv` > but we also have `pip` equivalent of it (for example you have a sequence of > `pip install -e` commands that you can run to get the same result as `uv > sync`). And it costs a bit - hatch_build.py in the root of Airflow is > unnecessarily complex to support it (with dynamic pre-installed packages > and some other dynamic code. We could get rid of it and replace the dynamic > stuff with static pyproject.toml in the root folder if we do not care about > `pip` installation for development. It would also simplify "quick start" > docs if we only support `uv` with workspaces. > > The problem with `uv` is that it's an open-source, but privately controlled > (astral.sh) by a VC-backed company. We know (personally) and like the > people behind `uv`, but at some future point in time, the ownership and > change of control might turn astral into not-that-open-source-friendly > (this is in a stark contrast with `pip`, `hatch`, `flit` - which are part > of the Python Software Foundation and under the Python Packaging Authority > working group. > > *Concerns* > > I personally have some reservations against exclusively supporting `uv`, > but I recognise that it makes our packaging unnecessarily complex if we > continue to support other workflows. It's great DevEX, fantastic > "Contributor Journey Optimisation" with `uv`. > > The risks, however, are small. If anything happens with `astral` - they > licence it with Apache 2 and MIT dual licensing, we - or anyone else - > could fork it as it happened with multiple other projects in the past > including Akka -> Pekko, Terraform -> OpenTofu and numerous others. > > And it does not affect our users. Packages published from us are > installable with anything (including pip, hatch, flit) because we follow > standards, it's only about the dev tooling. And - if all things fail, we > can always redevelop our own "glue" (like we already do in breeze) to allow > workspace or dependency groups to be easily usable. > > *Proposal* > > My proposal - for the sake of simplicity, better contributor's journey - > for now to drop `pip` support and have `uv` as the only supported dev > tooling. That would allow us to simplify docs and tooling support. > > I would love to hear what others think about it. > > J. >
