I am +1 on the idea and also see this as next logical step. Some
thoughts from my side:
(1) If we plan to split the distributions to separate api-server,
scheduler deployments in future - will we then also as next increment
the split into multiple logical uv.locks? Or will it be (for the time
being) keep a gloabal uv-lock for all and everything in the repo?
(2) I have a bit of worries if somebody in a PR adds a dependency and
the uv-lock is a brader upgrade - we need to ensure if uv-lock is
touched such PR needs to run all tests in all versions?
(3) is the uv.lock Python version dependent? Because we need to have a
lock file per Python version.
Other than this I hope that the change will be one that slims complexity
of dev tooling codebase with a minimal risk that we run into a uv-vendor
lock... but the way back is not blocked will just be effort if astral
turns into evil in the future.
Jens
On 19.08.25 16:12, Jarek Potiuk wrote:
Yeah. Dependabot has support for it and I am already testing it for a while
with breeze's `uv.lock` - so - to add a bit of context - I started slow by
commitiung `uv.lock` in breeze and seeing the kind of issues we might have.
One of the issues there is that I am not entirely sure if we want
dependabot to upgrade things for us for the whole "workspace", I find this
auto-upgrade that is happening currently way better - because it literally
gets us few hours feedback when things are broken by packages released by
3rd-parties - and we know when they break our tests, otherwise it's pretty
transparent.
If you look at our constraints history
https://github.com/apache/airflow/commits/constraints-main/ we have 3-4
constraints upgrades a DAY(!) and many of those upgrades are more than one
dependency. And this is really nice because those are literally
"checkpoints" of "dependencies we knew passed all the tests". So when
someone releases a dependency that breaks something we do not need to look
far - our "canary" build shows exactly what's changed since the last "good
canary run" - which means that we can identify and fix the problems super
quickly - because we know that this particular dependency upgrade caused
it. And it's super easy to test with breeze - you just reproduce it
failing, downgrade the dep and see it working - and you know 100% that's it.
This follows one of my favourite saying - "if something is painful, do it
more often" - this is why resurrecting and old branch (I am dreading a bit
the moment I will start preparing 2.11.1) is often quite painful - because
you see many things broken and you have to track it through 100s of
dependencies upgraded.
Current way Dependabot works will not give us the same experience though -
it's either going to be very annoying (with dependabot PR upgrade every
single day at the very least) - or causing longer investigations (if we
batch it for - say - weekly upgrades). None of those two is even close to
the current experience with constraints where things **just work** - until
they don't and then you know what to do.
So what's I am aiming at in this "uv.lock" change is not even to use
dependabot for it, but to fulfill those two seemingly contradictory goals:
* keep the seamless experience of upgrades in "canary" and allowing us to
diagnose and fix failures quickly
* but give even more stability and "standard" experience with `uv sync` and
`uv lock` features so that contributors and committers can get more
"stable" and more "works for me syndrom free" experience.
And I think what I proposed will allow us to eat cake and have it too -
what I proposed (without even having dependabot involved) will actually do
the job quite nicely on both fronts.
BTW. We will still likely use dependabot for airflow deps but **only** to
address security needs I think. We are already on a streak to squash and
remove all kinds of blockers to upgrade things, but having dependabot flag
it for us automatically would be quite nice.
J.
On Tue, Aug 19, 2025 at 2:48 PM Pavankumar Gopidesu <gopidesupa...@gmail.com>
wrote:
Thanks Jarek for bringing this up, I really like this idea we should use
uv.lock.
For upgrading packages in the uv.lock file, dependaboat started adding
support for this, but it seems there are quite a lot of issues.
Some i have seen dependaboat updates lock file but not updating the package
in toml file etc; there may be other issues similar to related. but I hope
these may be addressed at some point of time ( Not sure when :) )
https://docs.astral.sh/uv/guides/integration/dependency-bots/#dependabot
https://github.com/astral-sh/uv/issues/2512
https://github.com/dependabot/dependabot-core/issues?q=is%3Aissue%20state%3Aopen%20uv
Pavan
On Tue, Aug 19, 2025 at 11:18 AM Wei Lee <weilee...@gmail.com> wrote:
pip is not aiming and
will likely never be a "development/project management" tool and
I’m a bit surprised. I thought the dep group thing was for this.
3) to give our users a way to install released airflow in reproducible
way
be able to use `uv` export to pylock.toml and replace our
`--constraint`
installation.
I’m thinking of supporting both. Or at least for a while, till
`pylock.toml` becomes well-known (which might take a long long time…)
They’re generated files, so might not be that hard to maintain.
Best,
Wei
On Aug 19, 2025, at 5:45 PM, Jarek Potiuk <ja...@potiuk.com> wrote:
+1 to this idea. I considered whether we should use `pylock.toml`
instead, but it might not address the issue we intend to resolve based
on
my reading.
Yeah. pylock.toml and https://peps.python.org/pep-0751/ (in the
future)
are
better replacement for point 3:
3) to give our users a way to install released airflow in reproducible
way
This is the intention of pylock.toml - it should eventually allow our
users
to install airflow in a reproducible way so basically would replace the
constraints of ours. For now support for it is experimental in PyPI and
it
might change, also there is a general consensus that it does not fully
replace the ".lock" files feature for poetry, uv, pip-tools when it
comes
to local development, workspaces, monorepos (which don't even have
standardisation yet). Also `uv` supports PEP751 pylock.toml as a
(loosy -
it removes a number of metada kept in uv.lock specific for development
workflows) export format from its uv.lock.
My view (and also a lot of the information I got from packaging people
i
interact with) - that each of the "development/project management"
tools
(like uv/poetry etc.) will keep their own ".lock" format - simply
because
there is no possible way to encompass all the needs they have in
"generic
way". There was about 100 os pages discussion when PEP 751 was proposed
and
uv team took a very active part in it - but there seem to be a number
of
points that are not really "generic". For example pip is not aiming and
will likely never be a "development/project management" tool and it
does
not have a "workspace project" - it will always be really "end-user
installatiion tool" as it's primary function - and the result of this
different approach is what PEP 751 aims for - to allow reproducible
installation for end users.
So what I see as a natural step when we adopt `uv.lock` as development
"source of truth" and our "development/project management tool" - we
will
be able to use `uv` export to pylock.toml and replace our
`--constraint`
installation. But while several tools (both pip and uv) already support
exporting to the pylock.toml, there is no way YET to install packages
using
such lock to actually install packages. Pip for example has this issue
open
for it https://github.com/pypa/pip/issues/13334 and it waits for
someone
to volunteer and implement it. So for us it would make sense to switch
for
3) from constraints to pylock.toml - but only after `pip` and other way
people install airflow will be able to use it. Before that - there is
no
benefit of us using it. This might even stay like that: `uv.lock` for
development and pylock.toml published by us for users who want to
install
airflow in reproducible way. The `pylock.toml` has a number of benefits
comparing to current constraint approach - single lock supporting
multiple
python versions, cryptographic integrity and attestation support,
support
for multiple platforms, support for dependency groups, support for
default
groups - all that will make few of our "custom" mechanisms we
currently
"workaround" into the "solid features" of reproducible installation.
But
we
are not there yet :).\
J.
On Tue, Aug 19, 2025 at 10:59 AM Wei Lee <weilee...@gmail.com> wrote:
+1 to this idea. I considered whether we should use `pylock.toml`
instead,
but it might not address the issue we intend to resolve based on my
reading.
Best,
Wei
On Aug 19, 2025, at 5:51 AM, Ferruzzi, Dennis
<ferru...@amazon.com.INVALID> wrote:
I don't have any particular experience, but based just on what you
shared there, it does feel like a next logical step in the migration
to
these tools. Off-loading work and simplifying our CI is also always a
plus
in my opinion.
- ferruzzi
________________________________
From: Jarek Potiuk <ja...@potiuk.com>
Sent: Monday, August 18, 2025 4:14 AM
To: dev@airflow.apache.org
Subject: [EXT] [DISCUSS] Replace constraints with uv.lock mechanisms
for
dev env freeze
CAUTION: This email originated from outside of the organization. Do
not
click links or open attachments unless you can confirm the sender and
know
the content is safe.
AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur
externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si
vous
ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes
pas
certain que le contenu ne présente aucun risque.
Hello here,
*TL;DR: I would love to hear comments on the proposal to switch to
using
uv.lock for our dev env dependency freeze*
While working on some small follow-ups after prek upgrade, I think I
finally figured out how we can start using `uv.lock` locally and
merge
it
into our constraint mechanism effectively - without
over-burdening committers with merging a lot of "dependabot-like"
PRs.
The key problem that needed to be resolved was who and when would
update
the uv.lock regularly.
I captured my thoughts in
https://github.com/apache/airflow/issues/54609
But let me copy the content of it below. I would love to hear the
comments
- here or in the GitHub issue if you will (I will bring important
things
from the issue here if they will be raised there):
*Currently constraints are used for three purposes:*
1) to take snapshot of latest "working" version of dependencies
2) to limit PRs to "latest known good constraints" - so that newly
released
packages are not failing most PRs, but fail canary builds instead
3) to give our users a way to install released airflow in
reproducible
way
However, after we switched to uv, the case 2) can be better achieved
with
uv.lock. Currently we .gitignore uv.lock, but we could commit it to
the
repository and handle constraint generation differently
* constraints could be generated using latest lock
* canary builds could use the --resolution highest as today and
diverge
from latest "frozen" uv.lock
* all prs would use the uv.lock and do not use --resolution-highest
in
those PRs. This has the added benefit that uv sync in PR would
automatically upgrade dependencies if the dependencies in
pyproject.toml
are conflicting with uv.lock - but it would not use --resolution
highest,
which means that the PRs that modify dependeNcies do not have to get
latest
dependencies when they are run - that would limit the number of PRs
that
are experiencing "main/canary" problems
*Problem*
The only problem to solve is who and when would upgrade the uv.lock
and
commit it to the repo. Currently constraints are automatically bumped
after
successful canary runs, but they are in orphan un-protected branches,
while
uv.lock shares the code with main airflow repo, and we need a "human
with
ICLA" to review and merge commits there (ASF requirement)
*Solution*
Proposed solution is to add such a uv.lock upgrade to our
"upgrade-important-versions" prek hook. We run it in CI in canary
builds
and we fail the builds when new versions of important installers
change
(pip, Python base images, golang) - this happens often-enough to be
eventually-consistent pretty fast, but rarely enough to make it into
annoying "everyday" upgrade chore.
Regular PRs - especially those that would change dependencies will
also
modify uv.lock - for example when a new dependency is added or
removed
or
bumped. This means that the uv.lock will be generally quite regularly
bumped and "eventually consistent" with constraints.
*Benefits:*
* simplification of breeze build image with constraints (currently it
uses
complex uv pip install with --constraint, and it could simply use uv
sync
--all-packages instead if our uv.lock is checked in and maintained
* better synchronization of the local venv. Currently uv sync does
not
guarantee to have the same version of dependencies in local venv. The
only
way to guarantee it now is to use breeze image - build with
constraints
or
run complex commands with constraints added when you run uv sync
* less "3rd-party release" impact on failing PRs. if PRs will not
touch
the
dependency specification where the new release is released, it will
continue using the dependency stored in .lock and not the latest
dependency
released
*Cons:*
* Probably we should pay more attention to run
upgrade-important-versions
regularly and even add an option there to manually just perform the
sync
(for example that will be useful as part of the release procedure,
where
release manager would have to make sure to run the
upgrade-important-versions check when preparing release notes. But it
has
the advantage that it will be a conscious step that should actually
help
in
making sure that constraints are updated.
* sometimes PRs of people will have "big" changes to uv.lock - when
they
are updating the dependency and alongside the uv.lock will also bump
many
other unrelated dependencies - this could be mitigated by splitting
it
into
"lock upgrade" and "dependency update" but workflow of that is not
very
convenient and natural.
J.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
For additional commands, e-mail: dev-h...@airflow.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
For additional commands, e-mail: dev-h...@airflow.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
For additional commands, e-mail: dev-h...@airflow.apache.org
