This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v3-3-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/v3-3-test by this push:
new 29120eb4e72 [v3-3-test] Raise curated dependency floors in ci upgrade
runs (#73657) (#73668)
29120eb4e72 is described below
commit 29120eb4e72294b15b8d81f182d4b51ad2d86621
Author: Jarek Potiuk <[email protected]>
AuthorDate: Thu Sep 24 15:32:52 2026 +0200
[v3-3-test] Raise curated dependency floors in ci upgrade runs (#73657)
(#73668)
* Stop backporting the scheduled CI environment upgrade (#73319)
Each maintenance branch upgrades its own CI environment on its own
schedule, so backporting main's upgrade races that with a change that
conflicts on the very files it regenerates — which is why none of the
labelled upgrade PRs has ever produced a backport. boring-cyborg labels
by path and cannot tell this PR from a hand-written one touching dev/ or
.github/, so the label has to come off afterwards instead.
Generated-by: Claude Code (Opus 5)
(cherry picked from commit 613c9d3330d5f3989ef9bb854fa021808aa7ce4a)
* Raise curated dependency floors in ci upgrade runs (#73657)
* Add dependency floor policy configuration
Generated-by: Claude Opus 5
* Find curated dependency requirements and held-back packages
Generated-by: Claude Opus 5
* Select dependency floor targets outside the cooldown
Generated-by: Claude Opus 5
* Rewrite dependency floors in pyproject files
Generated-by: Claude Opus 5
* Roll back dependency floor bumps that break resolution
Generated-by: Claude Opus 5
* Add upgrade-dependency-floors manual hook
Generated-by: Claude Opus 5
* Raise curated dependency floors in breeze ci upgrade
Generated-by: Claude Opus 5
* Document automatic dependency floor bumps
Generated-by: Claude Opus 5
* Harden dependency floor bumps after review
Floors of workspace members with their own uv.lock (dev/breeze) are left
alone, since the root resolve check does not cover that lock. A rollback
restores only the occurrences a bump changed, a bump applies to all its
files or none, malformed PyPI data skips only the affected package, an
aborted run still leaves a note in the PR body, and an already-open
upgrade PR gets the body of the latest run.
Generated-by: Claude Opus 5
* Address remaining review notes on dependency floor bumps
Maintainers read the upgrade PR to see what moved, so the report now
names the files behind each raised floor. Caps set only in the root
[tool.uv] tables are honoured, an unreachable PyPI no longer stalls the
run for every curated package, and the report's temporary directory is
cleaned up.
Generated-by: Claude Opus 5
(cherry picked from commit 7bf71cd3eb820c103cb9566033d21d1aa3cef0fc)
---
.github/workflows/upgrade-check.yml | 23 +
.pre-commit-config.yaml | 12 +
dev/breeze/doc/08_ci_tasks.rst | 9 +
.../0018-raise-dependency-floors-automatically.md | 232 +++++++
dev/breeze/doc/images/output_ci_upgrade.svg | 44 +-
dev/breeze/doc/images/output_ci_upgrade.txt | 2 +-
.../src/airflow_breeze/commands/ci_commands.py | 179 ++++--
.../airflow_breeze/commands/ci_commands_config.py | 1 +
dev/breeze/tests/test_ci_upgrade.py | 111 ++++
dev/breeze/tests/test_remove_backport_labels.py | 85 +++
providers/amazon/pyproject.toml | 6 +-
pyproject.toml | 32 +
scripts/ci/prek/upgrade_dependency_floors.py | 537 +++++++++++++++++
.../ci/prek/test_upgrade_dependency_floors.py | 667 +++++++++++++++++++++
14 files changed, 1885 insertions(+), 55 deletions(-)
diff --git a/.github/workflows/upgrade-check.yml
b/.github/workflows/upgrade-check.yml
index eb9556f8919..106c304a11c 100644
--- a/.github/workflows/upgrade-check.yml
+++ b/.github/workflows/upgrade-check.yml
@@ -99,6 +99,29 @@ jobs:
--json url \
--jq '.[0].url' 2>/dev/null || true)
echo "pr-url=${PR_URL}" >> "${GITHUB_OUTPUT}"
+ - name: >-
+ [${{ inputs.target-branch }}] Drop backport labels
+ # breeze already removed these, but boring-cyborg labels on a webhook
and can land
+ # after it exits. Each branch upgrades itself on its own schedule, so
a backport
+ # would race that with a change conflicting on the files it
regenerates.
+ if: steps.find-pr.outputs.pr-url != ''
+ env:
+ PR_URL: ${{ steps.find-pr.outputs.pr-url }}
+ run: |
+ sleep 30
+ mapfile -t LABELS < <(gh pr view "${PR_URL}" \
+ --json labels \
+ --jq '.labels[].name | select(startswith("backport-to-"))')
+ if [[ ${#LABELS[@]} -eq 0 ]]; then
+ echo "No backport labels to remove."
+ exit 0
+ fi
+ ARGS=()
+ for label in "${LABELS[@]}"; do
+ ARGS+=(--remove-label "${label}")
+ done
+ gh pr edit "${PR_URL}" "${ARGS[@]}"
+ echo "Removed: ${LABELS[*]}"
- name: >-
[${{ inputs.target-branch }}] Notify Slack on success
if: success() && steps.find-pr.outputs.pr-url != ''
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index 196cc906d16..96c7cfd8860 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -362,6 +362,18 @@ repos:
^scripts/ci/prek/upgrade_important_versions\.py$
pass_filenames: false
require_serial: true
+ - id: upgrade-dependency-floors
+ name: Upgrade curated dependency floors (manual)
+ entry: ./scripts/ci/prek/upgrade_dependency_floors.py
+ stages: ['manual']
+ language: python
+ files: >
+ (?x)
+ ^pyproject\.toml$|
+ ^scripts/ci/prek/upgrade_dependency_floors\.py$
+ pass_filenames: false
+ require_serial: true
+ additional_dependencies: ['packaging>=25', 'requests>=2.31.0',
'rich>=13.6.0', 'tomli>=2.0.1']
- repo: https://github.com/adamchainz/blacken-docs
rev: fda77690955e9b63c6687d8806bafd56a526e45f # frozen: 1.20.0
hooks:
diff --git a/dev/breeze/doc/08_ci_tasks.rst b/dev/breeze/doc/08_ci_tasks.rst
index 747491107fe..9d3e78aba9f 100644
--- a/dev/breeze/doc/08_ci_tasks.rst
+++ b/dev/breeze/doc/08_ci_tasks.rst
@@ -41,6 +41,15 @@ components. It can automatically create a pull request with
the changes.
The command checks if you are on the correct branch (main or a version test
branch like v2-10-test) with a
clean repository. If not, it will offer to reset your repository to the latest
state from apache/airflow.
+The upgrade also raises the lower bounds of a curated set of fast-moving
dependencies (boto3,
+botocore, Google Cloud and Azure SDKs). The policy lives in
``[tool.airflow.dependency-floors]`` of the
+root ``pyproject.toml``: a floor is raised to the newest release that is at
least ``min-age`` (180 days)
+old, members of a group share one floor, and packages that are capped anywhere
or listed under
+``exclude`` are never touched. Bumps that make ``uv lock`` fail - with the
highest or the lowest-direct
+resolution - are rolled back and listed in the PR description together with
everything that was
+raised or skipped. Use ``--no-upgrade-dependency-floors`` to skip the step.
The design is recorded in
+`ADR 0018 <adr/0018-raise-dependency-floors-automatically.md>`_.
+
These are all available flags of ``upgrade`` command:
.. image:: ./images/output_ci_upgrade.svg
diff --git a/dev/breeze/doc/adr/0018-raise-dependency-floors-automatically.md
b/dev/breeze/doc/adr/0018-raise-dependency-floors-automatically.md
new file mode 100644
index 00000000000..0248de9ee4a
--- /dev/null
+++ b/dev/breeze/doc/adr/0018-raise-dependency-floors-automatically.md
@@ -0,0 +1,232 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+ -->
+
+<!-- START doctoc generated TOC please keep comment here to allow auto update
-->
+<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
+
+- [18. Raise the floors of fast-moving dependencies
automatically](#18-raise-the-floors-of-fast-moving-dependencies-automatically)
+ - [Status](#status)
+ - [Context](#context)
+ - [Decision](#decision)
+ - [Consequences](#consequences)
+
+<!-- END doctoc generated TOC please keep comment here to allow auto update -->
+
+# 18. Raise the floors of fast-moving dependencies automatically
+
+Date: 2026-09-24
+
+## Status
+
+Proposed
+
+## Context
+
+Lower bounds (`>=`) of external dependencies are only ever raised by hand.
Nothing in the
+automated `breeze ci upgrade` flow touches them, and
`check-dependency-lower-bounds` only
+requires that a floor exists, not that it is recent. Fast-moving SDKs drift
far behind: the
+amazon provider still declares `boto3>=1.41.0` despite a comment asking to
raise it
+"regularly". Ancient floors mean:
+
+- the resolver has a large candidate space to backtrack through (boto3
publishes daily);
+- the lowest-direct dependency tests exercise versions nobody runs any more,
and break when
+ those old versions stop installing or stop matching newer transitive deps.
+
+## Decision
+
+`breeze ci upgrade` (the "Upgrade important CI environment" run) gets a step
that raises the
+lower bounds of a curated set of dependencies. It follows these rules:
+
+- Raise the floors of a curated set of dependencies automatically, as part of
the regular
+ "Upgrade important CI environment" run.
+- Never raise a floor to a version released less than 6 months ago.
+- Never propose a floor that makes the workspace unresolvable (highest or
lowest-direct).
+- Never touch a package we are deliberately holding back.
+
+### Configuration
+
+New section in the root `pyproject.toml`, next to
`[tool.uv.exclude-newer-package]`, so the
+dependency policy lives in one place:
+
+```toml
+[tool.airflow.dependency-floors]
+# A floor is never raised to a release uploaded more recently than this.
+min-age = "180 days"
+# Curated packages whose floors are kept fresh. fnmatch-style globs on
canonical names.
+packages = [
+ "boto3",
+ "botocore",
+ "google-cloud-*",
+ "google-api-python-client",
+ "azure-*",
+]
+# Packages that must always share the same floor.
+groups = [
+ ["boto3", "botocore"],
+]
+
+[tool.airflow.dependency-floors.exclude]
+# package = "reason (link)"
+sagemaker-studio = "Do not change without approval from AWS
(providers/amazon/pyproject.toml)"
+pandas = "DataFrame XComs need pandas<3 (#70791)"
+pymysql = "Capped below 1.2 in PyPI constraints generation (#67491)"
+fastapi = "Repeated breakage on new releases (#59681, #59710, #59856, #61579,
#68578)"
+deltalake = "Missing ARM wheels on new releases (#59977, #60098, #60376)"
+gunicorn = "API server startup deadlocks above 25.1 (#62524)"
+qdrant-client = "Capped (#62193)"
+airbyte-api = "1.x breaks provider tests (#69081)"
+mysql-connector-python = "Releases without wheels for >=3.12 (#60889, #66026)"
+anthropic = "Provider SDK migration is version-sensitive (#72072, #72094)"
+azure-ai-projects = "2.5+ needs openai>=3, broke main (#73621, #73627)"
+```
+
+`min-age` accepts the same duration syntax as uv's `exclude-newer-package` (`N
days`,
+`N hours`, `N minutes`).
+
+The explicit `exclude` list is seeded from the caps/exclusions and the single
reverted
+dependency upgrade in the history since 2025-03 (`git log --grep
'^(Cap|Limit|Pin|Exclude)'`).
+Entries that are also caught by automatic exclusion are kept anyway: they
document *why*, and
+survive the cap being removed.
+
+### Component: `scripts/ci/prek/upgrade_dependency_floors.py`
+
+A uv inline-script (same header style as `check_dependency_lower_bounds.py`),
exposed as a
+manual-stage prek hook `upgrade-dependency-floors` (`pass_filenames: false`,
+`require_serial: true`). Units, each independently testable:
+
+1. **`load_config(root_pyproject) -> FloorConfig`** — parses the section
above; validates
+ that every group member is covered by `packages`.
+2. **`find_requirements(workspace) -> list[RequirementSite]`** — for each
workspace member's
+ `pyproject.toml` (members from `[tool.uv.workspace]`, reusing the logic in
+ `check_dependency_lower_bounds.get_workspace_distribution_names`), every
requirement in
+ `project.dependencies`, `project.optional-dependencies` and
`dependency-groups`, with file,
+ section and the raw string. Workspace distributions and URL requirements
are ignored. Members with a
+ `uv.lock` of their own (`dev/breeze`) are not scanned: the resolve check
covers only the
+ root lock, so raising their floors would leave their lock stale.
+3. **`get_exclusion_reason(package, sites, config) -> str | None`** — returns
a reason when the
+ package is in `exclude`, or when any site constrains it with `<`, `<=`,
`!=`, `==`, `~=`
+ or `===` ("held back by `<spec>` in `<file>`"). Caps in the root `[tool.uv]`
+ `constraint-dependencies` and `override-dependencies` count as well.
Otherwise `None`.
+4. **`find_target_version(releases, min_age, now) -> Version | None`** —
newest final
+ (non-pre, non-dev), non-yanked release whose earliest upload time is `<=
now - min_age`.
+ `now` is a parameter, so tests need no clock mocking. PyPI JSON
+ (`https://pypi.org/pypi/<name>/json`) is fetched once per package; a fetch
failure or
+ malformed release data skips the package with a reason instead of failing
the run. After
+ three consecutive connection failures PyPI is treated as unreachable and
the remaining
+ packages are skipped without further requests.
+5. **`find_group_target(...)`** — for a group, the newest version that
satisfies (4) for
+ *every* member; if none, the group is skipped.
+6. **`rewrite_requirement(raw, target) -> str | None`** — replaces the
`>=`/`>` value only
+ when `target` is higher, preserving extras, markers, other specifiers and
the original
+ quoting/formatting of the line. Returns `None` when there is nothing to
raise. Entries
+ split by marker (e.g. a higher floor for `python_version >= '3.14'`) are
handled per
+ entry: each is raised only if the target exceeds its current floor.
+7. **`resolve_check(workspace) -> ResolveResult`** — runs `uv lock --dry-run`
and
+ `uv lock --dry-run --resolution lowest-direct`; returns success or the
resolver stderr.
+ Nothing is written.
+8. **`apply_with_rollback(bumps)`** — applies all bumps (per package/group),
runs (7); on
+ failure bisects over the package/group units to find the failing ones,
rolls those back,
+ and re-checks until green. Each rolled-back unit carries the resolver
error. A bump is
+ applied to all of its files or to none, and a rollback restores only the
occurrences the
+ bump changed, so an identical requirement already at the target is never
lowered.
+
+Edits are text-level on the `pyproject.toml` files (replace the exact
requirement string) so
+comments and layout are preserved; provider `pyproject.toml` regeneration
already preserves
+dependency lists.
+
+#### Output
+
+- Console summary via `rich`.
+- A Markdown report written to the path in `DEPENDENCY_FLOORS_REPORT` (if set)
with three
+ sections: **Raised** (`package: old → new`, with files), **Skipped**
(reason),
+ **Rolled back** (resolver error, first lines).
+- Exit code 0 whenever the run completes (a rolled-back bump is a reported
outcome, not an
+ error), non-zero only for configuration errors or a lock that is broken
before any bump. In
+ that case the report says "Not updated: <error>", so the failure shows in
the PR.
+
+### Integration: `breeze ci upgrade`
+
+- New flag `--upgrade-dependency-floors/--no-upgrade-dependency-floors`,
default on, added to
+ the command's option groups in the config file.
+- New step `upgrade-dependency-floors` in `UPGRADE_COMMANDS`, **after**
+ `upgrade-important-versions` and **before** `update-uv-lock`, so `uv lock
--upgrade`
+ incorporates the new floors.
+- breeze sets `DEPENDENCY_FLOORS_REPORT` to a file in a temporary directory,
reads it (and
+ removes the directory) after the steps, and appends it to the PR body. The
upgrade branch
+ name is stable, so most runs update an already-open PR: its body is replaced
as well, so
+ the report always matches the pushed diff.
+
+### Error handling
+
+| Situation | Behaviour |
+|---|---|
+| PyPI fetch fails for a package, or its data is malformed | Skip it, reason
"PyPI metadata unavailable" |
+| Three consecutive connection failures | Skip the remaining packages, reason
"PyPI unreachable" |
+| No release old enough | Skip, reason "no release older than min-age" |
+| Floor already at/above target | Nothing to do; not listed |
+| Resolve check fails | Bisect, roll back offending units, report error |
+| Resolve check fails with *no* bumps applied | Abort the step with the error
(the lock was already broken) and leave files untouched |
+| Invalid config | Non-zero exit with a clear message, also written to the
report |
+
+### Testing
+
+`scripts/tests/ci/prek/test_upgrade_dependency_floors.py` (pytest,
parametrized):
+
+- target selection: min-age boundary, pre-releases/dev skipped, yanked
skipped, earliest
+ upload time used, none eligible;
+- group target: intersection across members, no common version;
+- rewrite: plain `>=`, `>` , with extras, with markers, with an extra `<` spec
present on
+ another entry, per-marker split floors, target not higher → `None`;
+- exclusion: explicit list, each capping operator, capped in one file only;
+- rollback: resolver mocked (`spec`/`autospec`) to fail for specific units;
verifies the
+ bisection rolls back exactly those and reports their errors; failure with no
bumps aborts.
+
+`dev/breeze/tests`: the new flag toggles the step, and the step runs between
+`upgrade-important-versions` and `update-uv-lock`.
+
+### Documentation
+
+- `dev/breeze/doc/08_ci_tasks.rst` ("Running ci upgrade"): describe floor
bumping, the
+ config section and the exclusion rules; regenerate the command images.
+- `providers/amazon/pyproject.toml`: replace "We should update minimum version
of boto3 …
+ regularly" with a pointer to the automation.
+- The config section itself carries short comments on `min-age`, `groups` and
`exclude`.
+
+No newsfragment: build/CI tooling. Individual floor bumps land in provider
changelogs through
+the normal `git log`-based release notes.
+
+
+## Consequences
+
+- Floors of the curated SDKs trail PyPI by roughly six months instead of
drifting for years,
+ so the resolver has fewer candidates to backtrack through and the
lowest-direct dependency
+ tests exercise versions people still run.
+- Raised floors reach users when the affected provider is next released; they
show up in the
+ provider changelog through the normal `git log`-based release notes.
+- A bump that breaks resolution never reaches the upgrade PR: it is rolled
back and reported
+ in the PR body, so it cannot block the lock refresh the way #73308 did.
+- Packages we hold back need no extra bookkeeping when they are capped;
uncapped ones that
+ must not move are listed in `[tool.airflow.dependency-floors.exclude]` with
a reason.
+- Out of scope:
+
+- Fixing the `constraints-version-check` slowdown: that job measures the lock
against the
+ newest PyPI release, which a 6-month-old floor never reaches. The lock
refresh
+ (`uv lock --upgrade`) remains the fix for that.
+- Bumping every external dependency. Coverage grows by extending the curated
list.
+- Lowering floors, adding caps, or editing `build-system.requires`.
diff --git a/dev/breeze/doc/images/output_ci_upgrade.svg
b/dev/breeze/doc/images/output_ci_upgrade.svg
index d5e86739e3b..371d20a7cd5 100644
--- a/dev/breeze/doc/images/output_ci_upgrade.svg
+++ b/dev/breeze/doc/images/output_ci_upgrade.svg
@@ -1,4 +1,4 @@
-<svg class="rich-terminal" viewBox="0 0 1482 1026.0"
xmlns="http://www.w3.org/2000/svg";>
+<svg class="rich-terminal" viewBox="0 0 1482 1099.2"
xmlns="http://www.w3.org/2000/svg";>
<!-- Generated with Rich https://www.textualize.io -->
<style>
@@ -43,7 +43,7 @@
<defs>
<clipPath id="breeze-ci-upgrade-clip-terminal">
- <rect x="0" y="0" width="1463.0" height="975.0" />
+ <rect x="0" y="0" width="1463.0" height="1048.2" />
</clipPath>
<clipPath id="breeze-ci-upgrade-line-0">
<rect x="0" y="1.5" width="1464" height="24.65"/>
@@ -162,9 +162,18 @@
<clipPath id="breeze-ci-upgrade-line-38">
<rect x="0" y="928.7" width="1464" height="24.65"/>
</clipPath>
+<clipPath id="breeze-ci-upgrade-line-39">
+ <rect x="0" y="953.1" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-ci-upgrade-line-40">
+ <rect x="0" y="977.5" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-ci-upgrade-line-41">
+ <rect x="0" y="1001.9" width="1464" height="24.65"/>
+ </clipPath>
</defs>
- <rect fill="#292929" stroke="rgba(255,255,255,0.35)" stroke-width="1"
x="1" y="1" width="1480" height="1024" rx="8"/><text
class="breeze-ci-upgrade-title" fill="#c5c8c6" text-anchor="middle" x="740"
y="27">Command: ci upgrade</text>
+ <rect fill="#292929" stroke="rgba(255,255,255,0.35)" stroke-width="1"
x="1" y="1" width="1480" height="1097.2" rx="8"/><text
class="breeze-ci-upgrade-title" fill="#c5c8c6" text-anchor="middle" x="740"
y="27">Command: ci upgrade</text>
<g transform="translate(26,22)">
<circle cx="0" cy="0" r="7" fill="#ff5f57"/>
<circle cx="22" cy="0" r="7" fill="#febc2e"/>
@@ -201,19 +210,22 @@
</text><text class="breeze-ci-upgrade-r5" x="0" y="605.6" textLength="12.2"
clip-path="url(#breeze-ci-upgrade-line-24)">│</text><text
class="breeze-ci-upgrade-r4" x="24.4" y="605.6" textLength="12.2"
clip-path="url(#breeze-ci-upgrade-line-24)">s</text><text
class="breeze-ci-upgrade-r1" x="744.2" y="605.6" textLength="158.6"
clip-path="url(#breeze-ci-upgrade-line-24)">dependencies </text><text
class="breeze-ci-upgrade-r5" x="902.8" y="605.6" textLength="439.2"
clip-path="url(#breeze- [...]
</text><text class="breeze-ci-upgrade-r5" x="0" y="630" textLength="12.2"
clip-path="url(#breeze-ci-upgrade-line-25)">│</text><text
class="breeze-ci-upgrade-r4" x="24.4" y="630" textLength="341.6"
clip-path="url(#breeze-ci-upgrade-line-25)">--upgrade-important-versions</text><text
class="breeze-ci-upgrade-r1" x="366" y="630" textLength="12.2"
clip-path="url(#breeze-ci-upgrade-line-25)">/</text><text
class="breeze-ci-upgrade-r4" x="378.2" y="630" textLength="341.6"
clip-path="url(#breeze- [...]
</text><text class="breeze-ci-upgrade-r5" x="0" y="654.4" textLength="12.2"
clip-path="url(#breeze-ci-upgrade-line-26)">│</text><text
class="breeze-ci-upgrade-r4" x="24.4" y="654.4" textLength="36.6"
clip-path="url(#breeze-ci-upgrade-line-26)">ons</text><text
class="breeze-ci-upgrade-r1" x="744.2" y="654.4" textLength="109.8"
clip-path="url(#breeze-ci-upgrade-line-26)">versions </text><text
class="breeze-ci-upgrade-r5" x="854" y="654.4" textLength="451.4"
clip-path="url(#breeze-ci-u [...]
-</text><text class="breeze-ci-upgrade-r5" x="0" y="678.8" textLength="12.2"
clip-path="url(#breeze-ci-upgrade-line-27)">│</text><text
class="breeze-ci-upgrade-r4" x="24.4" y="678.8" textLength="195.2"
clip-path="url(#breeze-ci-upgrade-line-27)">--update-uv-lock</text><text
class="breeze-ci-upgrade-r1" x="219.6" y="678.8" textLength="12.2"
clip-path="url(#breeze-ci-upgrade-line-27)">/</text><text
class="breeze-ci-upgrade-r4" x="231.8" y="678.8" textLength="231.8"
clip-path="url(#breeze-ci [...]
-</text><text class="breeze-ci-upgrade-r5" x="0" y="703.2" textLength="12.2"
clip-path="url(#breeze-ci-upgrade-line-28)">│</text><text
class="breeze-ci-upgrade-r1" x="744.2" y="703.2" textLength="695.4"
clip-path="url(#breeze-ci-upgrade-line-28)">resolutions inside Breeze CI image, then regenerate the  </text><text
class="breeze-ci-upgrade-r5" x="1451.8" y="703.2" textLength="12.2"
clip-path="url(#breeze-ci-upgrade-line-28)">│</text><text class [...]
-</text><text class="breeze-ci-upgrade-r5" x="0" y="727.6" textLength="12.2"
clip-path="url(#breeze-ci-upgrade-line-29)">│</text><text
class="breeze-ci-upgrade-r1" x="744.2" y="727.6" textLength="695.4"
clip-path="url(#breeze-ci-upgrade-line-29)">API datamodels the resolved code generator stamps its    </text><text
class="breeze-ci-upgrade-r5" x="1451.8" y="727.6" textLength="12.2"
clip-path="url(#breeze-ci-upgrade-line-29)">│</text>< [...]
-</text><text class="breeze-ci-upgrade-r5" x="0" y="752" textLength="12.2"
clip-path="url(#breeze-ci-upgrade-line-30)">│</text><text
class="breeze-ci-upgrade-r1" x="744.2" y="752" textLength="158.6"
clip-path="url(#breeze-ci-upgrade-line-30)">version into </text><text
class="breeze-ci-upgrade-r5" x="902.8" y="752" textLength="305"
clip-path="url(#breeze-ci-upgrade-line-30)">[default: update-uv-lock]</text><text
class="breeze-ci-upgrade-r5" x="1451.8" y="752" textLength="12. [...]
-</text><text class="breeze-ci-upgrade-r5" x="0" y="776.4" textLength="12.2"
clip-path="url(#breeze-ci-upgrade-line-31)">│</text><text
class="breeze-ci-upgrade-r4" x="24.4" y="776.4" textLength="207.4"
clip-path="url(#breeze-ci-upgrade-line-31)">--k8s-schema-sync</text><text
class="breeze-ci-upgrade-r1" x="231.8" y="776.4" textLength="12.2"
clip-path="url(#breeze-ci-upgrade-line-31)">/</text><text
class="breeze-ci-upgrade-r4" x="244" y="776.4" textLength="244"
clip-path="url(#breeze-ci-up [...]
-</text><text class="breeze-ci-upgrade-r5" x="0" y="800.8" textLength="12.2"
clip-path="url(#breeze-ci-upgrade-line-32)">│</text><text
class="breeze-ci-upgrade-r5" x="744.2" y="800.8" textLength="195.2"
clip-path="url(#breeze-ci-upgrade-line-32)">k8s-schema-sync]</text><text
class="breeze-ci-upgrade-r5" x="1451.8" y="800.8" textLength="12.2"
clip-path="url(#breeze-ci-upgrade-line-32)">│</text><text
class="breeze-ci-upgrade-r1" x="1464" y="800.8" textLength="12.2"
clip-path="url(#breeze-ci [...]
-</text><text class="breeze-ci-upgrade-r5" x="0" y="825.2" textLength="1464"
clip-path="url(#breeze-ci-upgrade-line-33)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text
class="breeze-ci-upgrade-r1" x="1464" y="825.2" textLength="12.2"
clip-path="url(#breeze-ci-upgrade-line-33)">
-</text><text class="breeze-ci-upgrade-r5" x="0" y="849.6" textLength="24.4"
clip-path="url(#breeze-ci-upgrade-line-34)">╭─</text><text
class="breeze-ci-upgrade-r5" x="24.4" y="849.6" textLength="195.2"
clip-path="url(#breeze-ci-upgrade-line-34)"> Common options </text><text
class="breeze-ci-upgrade-r5" x="219.6" y="849.6" textLength="1220"
clip-path="url(#breeze-ci-upgrade-line-34)">───────────────────────────────────────────────────────────────────────────────────────────
[...]
-</text><text class="breeze-ci-upgrade-r5" x="0" y="874" textLength="12.2"
clip-path="url(#breeze-ci-upgrade-line-35)">│</text><text
class="breeze-ci-upgrade-r4" x="24.4" y="874" textLength="109.8"
clip-path="url(#breeze-ci-upgrade-line-35)">--answer </text><text
class="breeze-ci-upgrade-r7" x="158.6" y="874" textLength="24.4"
clip-path="url(#breeze-ci-upgrade-line-35)">-a</text><text
class="breeze-ci-upgrade-r1" x="207.4" y="874" textLength="329.4"
clip-path="url(#breeze-ci-upgrade- [...]
-</text><text class="breeze-ci-upgrade-r5" x="0" y="898.4" textLength="12.2"
clip-path="url(#breeze-ci-upgrade-line-36)">│</text><text
class="breeze-ci-upgrade-r4" x="24.4" y="898.4" textLength="109.8"
clip-path="url(#breeze-ci-upgrade-line-36)">--verbose</text><text
class="breeze-ci-upgrade-r7" x="158.6" y="898.4" textLength="24.4"
clip-path="url(#breeze-ci-upgrade-line-36)">-v</text><text
class="breeze-ci-upgrade-r1" x="207.4" y="898.4" textLength="585.6"
clip-path="url(#breeze-ci-upgra [...]
-</text><text class="breeze-ci-upgrade-r5" x="0" y="922.8" textLength="12.2"
clip-path="url(#breeze-ci-upgrade-line-37)">│</text><text
class="breeze-ci-upgrade-r4" x="24.4" y="922.8" textLength="109.8"
clip-path="url(#breeze-ci-upgrade-line-37)">--dry-run</text><text
class="breeze-ci-upgrade-r7" x="158.6" y="922.8" textLength="24.4"
clip-path="url(#breeze-ci-upgrade-line-37)">-D</text><text
class="breeze-ci-upgrade-r1" x="207.4" y="922.8" textLength="719.8"
clip-path="url(#breeze-ci-upgra [...]
-</text><text class="breeze-ci-upgrade-r5" x="0" y="947.2" textLength="12.2"
clip-path="url(#breeze-ci-upgrade-line-38)">│</text><text
class="breeze-ci-upgrade-r4" x="24.4" y="947.2" textLength="109.8"
clip-path="url(#breeze-ci-upgrade-line-38)">--help   </text><text
class="breeze-ci-upgrade-r7" x="158.6" y="947.2" textLength="24.4"
clip-path="url(#breeze-ci-upgrade-line-38)">-h</text><text
class="breeze-ci-upgrade-r1" x="207.4" y="947.2" textLength="329.4"
clip-path="url(# [...]
-</text><text class="breeze-ci-upgrade-r5" x="0" y="971.6" textLength="1464"
clip-path="url(#breeze-ci-upgrade-line-39)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text
class="breeze-ci-upgrade-r1" x="1464" y="971.6" textLength="12.2"
clip-path="url(#breeze-ci-upgrade-line-39)">
+</text><text class="breeze-ci-upgrade-r5" x="0" y="678.8" textLength="12.2"
clip-path="url(#breeze-ci-upgrade-line-27)">│</text><text
class="breeze-ci-upgrade-r4" x="24.4" y="678.8" textLength="329.4"
clip-path="url(#breeze-ci-upgrade-line-27)">--upgrade-dependency-floors</text><text
class="breeze-ci-upgrade-r1" x="353.8" y="678.8" textLength="12.2"
clip-path="url(#breeze-ci-upgrade-line-27)">/</text><text
class="breeze-ci-upgrade-r4" x="366" y="678.8" textLength="353.8"
clip-path="url(# [...]
+</text><text class="breeze-ci-upgrade-r5" x="0" y="703.2" textLength="12.2"
clip-path="url(#breeze-ci-upgrade-line-28)">│</text><text
class="breeze-ci-upgrade-r4" x="24.4" y="703.2" textLength="12.2"
clip-path="url(#breeze-ci-upgrade-line-28)">s</text><text
class="breeze-ci-upgrade-r1" x="744.2" y="703.2" textLength="402.6"
clip-path="url(#breeze-ci-upgrade-line-28)">[tool.airflow.dependency-floors] </text><text
class="breeze-ci-upgrade-r5" x="1146.8" y="703.2" textLength="122" clip [...]
+</text><text class="breeze-ci-upgrade-r5" x="0" y="727.6" textLength="12.2"
clip-path="url(#breeze-ci-upgrade-line-29)">│</text><text
class="breeze-ci-upgrade-r5" x="744.2" y="727.6" textLength="317.2"
clip-path="url(#breeze-ci-upgrade-line-29)">upgrade-dependency-floors]</text><text
class="breeze-ci-upgrade-r5" x="1451.8" y="727.6" textLength="12.2"
clip-path="url(#breeze-ci-upgrade-line-29)">│</text><text
class="breeze-ci-upgrade-r1" x="1464" y="727.6" textLength="12.2"
clip-path="url( [...]
+</text><text class="breeze-ci-upgrade-r5" x="0" y="752" textLength="12.2"
clip-path="url(#breeze-ci-upgrade-line-30)">│</text><text
class="breeze-ci-upgrade-r4" x="24.4" y="752" textLength="195.2"
clip-path="url(#breeze-ci-upgrade-line-30)">--update-uv-lock</text><text
class="breeze-ci-upgrade-r1" x="219.6" y="752" textLength="12.2"
clip-path="url(#breeze-ci-upgrade-line-30)">/</text><text
class="breeze-ci-upgrade-r4" x="231.8" y="752" textLength="231.8"
clip-path="url(#breeze-ci-upgrade [...]
+</text><text class="breeze-ci-upgrade-r5" x="0" y="776.4" textLength="12.2"
clip-path="url(#breeze-ci-upgrade-line-31)">│</text><text
class="breeze-ci-upgrade-r1" x="744.2" y="776.4" textLength="695.4"
clip-path="url(#breeze-ci-upgrade-line-31)">resolutions inside Breeze CI image, then regenerate the  </text><text
class="breeze-ci-upgrade-r5" x="1451.8" y="776.4" textLength="12.2"
clip-path="url(#breeze-ci-upgrade-line-31)">│</text><text class [...]
+</text><text class="breeze-ci-upgrade-r5" x="0" y="800.8" textLength="12.2"
clip-path="url(#breeze-ci-upgrade-line-32)">│</text><text
class="breeze-ci-upgrade-r1" x="744.2" y="800.8" textLength="695.4"
clip-path="url(#breeze-ci-upgrade-line-32)">API datamodels the resolved code generator stamps its    </text><text
class="breeze-ci-upgrade-r5" x="1451.8" y="800.8" textLength="12.2"
clip-path="url(#breeze-ci-upgrade-line-32)">│</text>< [...]
+</text><text class="breeze-ci-upgrade-r5" x="0" y="825.2" textLength="12.2"
clip-path="url(#breeze-ci-upgrade-line-33)">│</text><text
class="breeze-ci-upgrade-r1" x="744.2" y="825.2" textLength="158.6"
clip-path="url(#breeze-ci-upgrade-line-33)">version into </text><text
class="breeze-ci-upgrade-r5" x="902.8" y="825.2" textLength="305"
clip-path="url(#breeze-ci-upgrade-line-33)">[default: update-uv-lock]</text><text
class="breeze-ci-upgrade-r5" x="1451.8" y="825.2" textLen [...]
+</text><text class="breeze-ci-upgrade-r5" x="0" y="849.6" textLength="12.2"
clip-path="url(#breeze-ci-upgrade-line-34)">│</text><text
class="breeze-ci-upgrade-r4" x="24.4" y="849.6" textLength="207.4"
clip-path="url(#breeze-ci-upgrade-line-34)">--k8s-schema-sync</text><text
class="breeze-ci-upgrade-r1" x="231.8" y="849.6" textLength="12.2"
clip-path="url(#breeze-ci-upgrade-line-34)">/</text><text
class="breeze-ci-upgrade-r4" x="244" y="849.6" textLength="244"
clip-path="url(#breeze-ci-up [...]
+</text><text class="breeze-ci-upgrade-r5" x="0" y="874" textLength="12.2"
clip-path="url(#breeze-ci-upgrade-line-35)">│</text><text
class="breeze-ci-upgrade-r5" x="744.2" y="874" textLength="195.2"
clip-path="url(#breeze-ci-upgrade-line-35)">k8s-schema-sync]</text><text
class="breeze-ci-upgrade-r5" x="1451.8" y="874" textLength="12.2"
clip-path="url(#breeze-ci-upgrade-line-35)">│</text><text
class="breeze-ci-upgrade-r1" x="1464" y="874" textLength="12.2"
clip-path="url(#breeze-ci-upgrade [...]
+</text><text class="breeze-ci-upgrade-r5" x="0" y="898.4" textLength="1464"
clip-path="url(#breeze-ci-upgrade-line-36)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text
class="breeze-ci-upgrade-r1" x="1464" y="898.4" textLength="12.2"
clip-path="url(#breeze-ci-upgrade-line-36)">
+</text><text class="breeze-ci-upgrade-r5" x="0" y="922.8" textLength="24.4"
clip-path="url(#breeze-ci-upgrade-line-37)">╭─</text><text
class="breeze-ci-upgrade-r5" x="24.4" y="922.8" textLength="195.2"
clip-path="url(#breeze-ci-upgrade-line-37)"> Common options </text><text
class="breeze-ci-upgrade-r5" x="219.6" y="922.8" textLength="1220"
clip-path="url(#breeze-ci-upgrade-line-37)">───────────────────────────────────────────────────────────────────────────────────────────
[...]
+</text><text class="breeze-ci-upgrade-r5" x="0" y="947.2" textLength="12.2"
clip-path="url(#breeze-ci-upgrade-line-38)">│</text><text
class="breeze-ci-upgrade-r4" x="24.4" y="947.2" textLength="109.8"
clip-path="url(#breeze-ci-upgrade-line-38)">--answer </text><text
class="breeze-ci-upgrade-r7" x="158.6" y="947.2" textLength="24.4"
clip-path="url(#breeze-ci-upgrade-line-38)">-a</text><text
class="breeze-ci-upgrade-r1" x="207.4" y="947.2" textLength="329.4"
clip-path="url(#breeze-ci- [...]
+</text><text class="breeze-ci-upgrade-r5" x="0" y="971.6" textLength="12.2"
clip-path="url(#breeze-ci-upgrade-line-39)">│</text><text
class="breeze-ci-upgrade-r4" x="24.4" y="971.6" textLength="109.8"
clip-path="url(#breeze-ci-upgrade-line-39)">--verbose</text><text
class="breeze-ci-upgrade-r7" x="158.6" y="971.6" textLength="24.4"
clip-path="url(#breeze-ci-upgrade-line-39)">-v</text><text
class="breeze-ci-upgrade-r1" x="207.4" y="971.6" textLength="585.6"
clip-path="url(#breeze-ci-upgra [...]
+</text><text class="breeze-ci-upgrade-r5" x="0" y="996" textLength="12.2"
clip-path="url(#breeze-ci-upgrade-line-40)">│</text><text
class="breeze-ci-upgrade-r4" x="24.4" y="996" textLength="109.8"
clip-path="url(#breeze-ci-upgrade-line-40)">--dry-run</text><text
class="breeze-ci-upgrade-r7" x="158.6" y="996" textLength="24.4"
clip-path="url(#breeze-ci-upgrade-line-40)">-D</text><text
class="breeze-ci-upgrade-r1" x="207.4" y="996" textLength="719.8"
clip-path="url(#breeze-ci-upgrade-line- [...]
+</text><text class="breeze-ci-upgrade-r5" x="0" y="1020.4" textLength="12.2"
clip-path="url(#breeze-ci-upgrade-line-41)">│</text><text
class="breeze-ci-upgrade-r4" x="24.4" y="1020.4" textLength="109.8"
clip-path="url(#breeze-ci-upgrade-line-41)">--help   </text><text
class="breeze-ci-upgrade-r7" x="158.6" y="1020.4" textLength="24.4"
clip-path="url(#breeze-ci-upgrade-line-41)">-h</text><text
class="breeze-ci-upgrade-r1" x="207.4" y="1020.4" textLength="329.4"
clip-path="u [...]
+</text><text class="breeze-ci-upgrade-r5" x="0" y="1044.8" textLength="1464"
clip-path="url(#breeze-ci-upgrade-line-42)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text
class="breeze-ci-upgrade-r1" x="1464" y="1044.8" textLength="12.2"
clip-path="url(#breeze-ci-upgrade-line-42)">
</text>
</g>
</g>
diff --git a/dev/breeze/doc/images/output_ci_upgrade.txt
b/dev/breeze/doc/images/output_ci_upgrade.txt
index 0dfdad6effa..412c3bbf03f 100644
--- a/dev/breeze/doc/images/output_ci_upgrade.txt
+++ b/dev/breeze/doc/images/output_ci_upgrade.txt
@@ -1 +1 @@
-edaa8a170a17c7208ed46a813e40aa57
+f90a9e29f581253c78e725f021dee69c
diff --git a/dev/breeze/src/airflow_breeze/commands/ci_commands.py
b/dev/breeze/src/airflow_breeze/commands/ci_commands.py
index 70c5fa01f37..ec4ec346a80 100644
--- a/dev/breeze/src/airflow_breeze/commands/ci_commands.py
+++ b/dev/breeze/src/airflow_breeze/commands/ci_commands.py
@@ -535,6 +535,121 @@ def _sync_k8s_schemas_to_airflow_site(airflow_site: Path,
force: bool, command_e
run_command(cmd, check=False, env=command_env)
+def remove_backport_labels(*, branch_name: str, command_env: dict[str, str])
-> None:
+ """Drop any ``backport-to-*`` label boring-cyborg put on the upgrade PR.
+
+ The CI environment is upgraded on each maintenance branch by that branch's
own scheduled
+ run, so backporting main's upgrade would race it with a change that
conflicts on the very
+ files it regenerates. boring-cyborg labels by path and cannot tell this PR
apart from a
+ hand-written one touching ``dev/`` or ``.github/``, so the label is
removed here instead.
+
+ Labelling happens on a webhook and may land after this runs;
``upgrade-check.yml`` sweeps
+ again once the PR has settled, so a miss here is not the last word.
+ """
+ labels_result = run_command(
+ [
+ "gh",
+ "pr",
+ "view",
+ branch_name,
+ "--repo",
+ "apache/airflow",
+ "--json",
+ "labels",
+ "--jq",
+ '[.labels[].name | select(startswith("backport-to-"))] |
join(",")',
+ ],
+ capture_output=True,
+ text=True,
+ check=False,
+ env=command_env,
+ )
+ if labels_result.returncode != 0:
+ console_print("[warning]Could not read PR labels - leaving any
backport labels in place.[/]")
+ return
+ labels = [label for label in labels_result.stdout.strip().split(",") if
label]
+ if not labels:
+ return
+ remove_cmd = ["gh", "pr", "edit", branch_name, "--repo", "apache/airflow"]
+ for label in labels:
+ remove_cmd.extend(["--remove-label", label])
+ if run_command(remove_cmd, capture_output=True, text=True, check=False,
env=command_env).returncode:
+ console_print(f"[warning]Could not remove backport labels: {',
'.join(labels)}[/]")
+ else:
+ console_print(f"[success]Removed backport labels: {',
'.join(labels)}.[/]")
+
+
+# All upgrade commands run locally with check=False to continue on errors.
+# The uv lock --upgrade step must run after the steps that change
dependencies, so it can
+# incorporate them -- and before the steps that regenerate code from the
resolved versions.
+UPGRADE_COMMANDS: list[tuple[str, str]] = [
+ ("autoupdate", "prek autoupdate --cooldown-days 4 --freeze"),
+ (
+ "update-chart-dependencies",
+ "prek --all-files --show-diff-on-failure --color always --verbose
--stage manual update-chart-dependencies",
+ ),
+ (
+ "upgrade-important-versions",
+ "prek --all-files --show-diff-on-failure --color always --verbose
--stage manual upgrade-important-versions",
+ ),
+ (
+ "upgrade-dependency-floors",
+ "prek --all-files --show-diff-on-failure --color always --verbose
--stage manual upgrade-dependency-floors",
+ ),
+ (
+ "update-uv-lock",
+ "uv lock --upgrade",
+ ),
+ (
+ # The lock upgrade can bump datamodel-code-generator, whose version is
stamped into the
+ # generated files' header. These hooks only watch the API sources,
which an upgrade run
+ # never touches, so nothing regenerates them here and CI's --all-files
run goes red.
+ "regenerate-datamodels",
+ "prek --all-files --show-diff-on-failure --color always --verbose "
+ "generate-tasksdk-datamodels generate-airflowctl-datamodels",
+ ),
+]
+
+
+DEPENDENCY_FLOORS_REPORT_ENV = "DEPENDENCY_FLOORS_REPORT"
+UPGRADE_PR_BODY = "This PR upgrades important dependencies of the CI
environment."
+
+
+def build_upgrade_pr_body(floors_report: str | None) -> str:
+ if not floors_report:
+ return UPGRADE_PR_BODY
+ return f"{UPGRADE_PR_BODY}\n\n{floors_report}"
+
+
+def get_step_enabled(
+ *,
+ autoupdate: bool,
+ update_chart_dependencies: bool,
+ upgrade_important_versions: bool,
+ upgrade_dependency_floors: bool,
+ update_uv_lock: bool,
+) -> dict[str, bool]:
+ return {
+ "autoupdate": autoupdate,
+ "update-chart-dependencies": update_chart_dependencies,
+ "upgrade-important-versions": upgrade_important_versions,
+ "upgrade-dependency-floors": upgrade_dependency_floors,
+ "update-uv-lock": update_uv_lock,
+ "regenerate-datamodels": update_uv_lock,
+ }
+
+
+def read_floors_report(report_path: Path) -> str | None:
+ """Return the report the floors step wrote, if any, and remove its
temporary directory."""
+ report = report_path.read_text() if report_path.exists() else None
+ shutil.rmtree(report_path.parent, ignore_errors=True)
+ return report
+
+
+def build_update_pr_body_command(branch_name: str, pr_body: str) -> list[str]:
+ return ["gh", "pr", "edit", branch_name, "--repo", "apache/airflow",
"--body", pr_body]
+
+
@ci_group.command(
name="upgrade",
help="Perform important upgrade steps of the CI environment. And create a
PR",
@@ -595,6 +710,12 @@ def _sync_k8s_schemas_to_airflow_site(airflow_site: Path,
force: bool, command_e
show_default=True,
help="Run upgrade-important-versions to bump key dependency versions",
)
[email protected](
+ "--upgrade-dependency-floors/--no-upgrade-dependency-floors",
+ default=True,
+ show_default=True,
+ help="Raise the floors of the curated dependencies in
[tool.airflow.dependency-floors]",
+)
@click.option(
"--update-uv-lock/--no-update-uv-lock",
default=True,
@@ -622,6 +743,7 @@ def upgrade(
autoupdate: bool,
update_chart_dependencies: bool,
upgrade_important_versions: bool,
+ upgrade_dependency_floors: bool,
update_uv_lock: bool,
k8s_schema_sync: bool,
github_token: str | None,
@@ -795,47 +917,24 @@ def upgrade(
"Commands may fail if they require authentication.[/]"
)
- # All upgrade commands run locally with check=False to continue on errors.
- # The uv lock --upgrade step must run after the steps that change
dependencies, so it can
- # incorporate them -- and before the steps that regenerate code from the
resolved versions.
- upgrade_commands: list[tuple[str, str]] = [
- ("autoupdate", "prek autoupdate --cooldown-days 4 --freeze"),
- (
- "update-chart-dependencies",
- "prek --all-files --show-diff-on-failure --color always --verbose
--stage manual update-chart-dependencies",
- ),
- (
- "upgrade-important-versions",
- "prek --all-files --show-diff-on-failure --color always --verbose
--stage manual upgrade-important-versions",
- ),
- (
- "update-uv-lock",
- "uv lock --upgrade",
- ),
- (
- # The lock upgrade can bump datamodel-code-generator, whose
version is stamped into the
- # generated files' header. These hooks only watch the API sources,
which an upgrade run
- # never touches, so nothing regenerates them here and CI's
--all-files run goes red.
- "regenerate-datamodels",
- "prek --all-files --show-diff-on-failure --color always --verbose "
- "generate-tasksdk-datamodels generate-airflowctl-datamodels",
- ),
- ]
+ floors_report_path = Path(tempfile.mkdtemp()) / "dependency-floors.md"
+ command_env[DEPENDENCY_FLOORS_REPORT_ENV] = str(floors_report_path)
- step_enabled = {
- "autoupdate": autoupdate,
- "update-chart-dependencies": update_chart_dependencies,
- "upgrade-important-versions": upgrade_important_versions,
- "update-uv-lock": update_uv_lock,
- "regenerate-datamodels": update_uv_lock,
- }
+ step_enabled = get_step_enabled(
+ autoupdate=autoupdate,
+ update_chart_dependencies=update_chart_dependencies,
+ upgrade_important_versions=upgrade_important_versions,
+ upgrade_dependency_floors=upgrade_dependency_floors,
+ update_uv_lock=update_uv_lock,
+ )
# Execute upgrade commands
- for step_name, command in upgrade_commands:
+ for step_name, command in UPGRADE_COMMANDS:
if step_enabled[step_name]:
run_command(command.split(), check=False, env=command_env)
else:
console_print(f"[info]Skipping {step_name} (disabled).[/]")
+ floors_report = read_floors_report(floors_report_path)
# Sync K8s schemas to airflow-site
if k8s_schema_sync:
@@ -915,7 +1014,7 @@ def upgrade(
console_print("[warning]Could not determine fork repository. Using
branch name only.[/]")
pr_title = f"[{target_branch}] Upgrade important CI environment"
- pr_body = "This PR upgrades important dependencies of the CI
environment."
+ pr_body = build_upgrade_pr_body(floors_report)
# Check if there's already an open PR for this branch.
# gh pr list / gh pr ready filter by the bare head-branch name, not the
@@ -948,6 +1047,14 @@ def upgrade(
if existing_pr and existing_pr != "null" and existing_pr != "":
console_print(f"[success]Existing PR found and updated with force
push: {existing_pr}[/]")
+ # The body carries this run's dependency-floors report, so it must
follow the pushed diff.
+ run_command(
+ build_update_pr_body_command(branch_name, pr_body),
+ capture_output=True,
+ text=True,
+ check=False,
+ env=command_env,
+ )
if draft:
# Convert back to draft so a human must undraft to trigger CI
run_command(
@@ -1004,6 +1111,8 @@ def upgrade(
else:
console_print(f"[success]PR created successfully:
{pr_result.stdout.strip()}.[/]")
+ remove_backport_labels(branch_name=branch_name,
command_env=command_env)
+
# Switch back to appropriate branch and delete the temporary branch
console_print(f"[info]Cleaning up temporary branch
{branch_name}...[/]")
if user_switched_to_target:
diff --git a/dev/breeze/src/airflow_breeze/commands/ci_commands_config.py
b/dev/breeze/src/airflow_breeze/commands/ci_commands_config.py
index ae5bc64a5b7..d8fdf9359bb 100644
--- a/dev/breeze/src/airflow_breeze/commands/ci_commands_config.py
+++ b/dev/breeze/src/airflow_breeze/commands/ci_commands_config.py
@@ -88,6 +88,7 @@ CI_PARAMETERS: dict[str, list[dict[str, str | list[str]]]] = {
"--autoupdate",
"--update-chart-dependencies",
"--upgrade-important-versions",
+ "--upgrade-dependency-floors",
"--update-uv-lock",
"--k8s-schema-sync",
],
diff --git a/dev/breeze/tests/test_ci_upgrade.py
b/dev/breeze/tests/test_ci_upgrade.py
new file mode 100644
index 00000000000..f44b5218b9b
--- /dev/null
+++ b/dev/breeze/tests/test_ci_upgrade.py
@@ -0,0 +1,111 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import pytest
+
+from airflow_breeze.commands.ci_commands import (
+ UPGRADE_COMMANDS,
+ build_update_pr_body_command,
+ build_upgrade_pr_body,
+ get_step_enabled,
+ read_floors_report,
+ upgrade,
+)
+
+STEP_NAMES = [name for name, _ in UPGRADE_COMMANDS]
+
+
+def test_floor_step_runs_after_important_versions_and_before_lock_upgrade():
+ assert (
+ STEP_NAMES.index("upgrade-important-versions")
+ < STEP_NAMES.index("upgrade-dependency-floors")
+ < STEP_NAMES.index("update-uv-lock")
+ )
+
+
+def test_floor_step_runs_the_manual_hook():
+ command = dict(UPGRADE_COMMANDS)["upgrade-dependency-floors"]
+ assert command.endswith("--stage manual upgrade-dependency-floors")
+
+
[email protected](
+ ("report", "expected_suffix"),
+ [
+ pytest.param(None, "environment.", id="no-report"),
+ pytest.param("", "environment.", id="empty-report"),
+ pytest.param("### Dependency floors\n\nRaised: none\n", "Raised:
none\n", id="report"),
+ ],
+)
+def test_build_upgrade_pr_body(report, expected_suffix):
+ body = build_upgrade_pr_body(report)
+ assert body.startswith("This PR upgrades important dependencies of the CI
environment.")
+ assert body.endswith(expected_suffix)
+
+
+def test_upgrade_has_floor_flag():
+ option = next(p for p in upgrade.params if p.name ==
"upgrade_dependency_floors")
+ assert option.default is True
+
+
+def test_existing_pr_body_is_replaced_with_the_new_report():
+ # The upgrade branch name is stable, so most runs update an open PR
instead of creating one.
+ assert build_update_pr_body_command("ci-upgrade-main", "body with report")
== [
+ "gh",
+ "pr",
+ "edit",
+ "ci-upgrade-main",
+ "--repo",
+ "apache/airflow",
+ "--body",
+ "body with report",
+ ]
+
+
+ALL_STEPS_ON = dict(
+ autoupdate=True,
+ update_chart_dependencies=True,
+ upgrade_important_versions=True,
+ upgrade_dependency_floors=True,
+ update_uv_lock=True,
+)
+
+
+def test_step_enabled_covers_every_step():
+ assert set(get_step_enabled(**ALL_STEPS_ON)) == set(STEP_NAMES)
+
+
+def test_no_upgrade_dependency_floors_disables_only_that_step():
+ enabled = get_step_enabled(**{**ALL_STEPS_ON, "upgrade_dependency_floors":
False})
+ assert [name for name, on in enabled.items() if not on] ==
["upgrade-dependency-floors"]
+
+
[email protected](
+ ("content", "expected"),
+ [
+ pytest.param("### Dependency floors\n", "### Dependency floors\n",
id="report"),
+ pytest.param(None, None, id="missing"),
+ ],
+)
+def test_read_floors_report_removes_its_temp_dir(tmp_path, content, expected):
+ report_dir = tmp_path / "floors"
+ report_dir.mkdir()
+ report_path = report_dir / "dependency-floors.md"
+ if content is not None:
+ report_path.write_text(content)
+ assert read_floors_report(report_path) == expected
+ assert not report_dir.exists()
diff --git a/dev/breeze/tests/test_remove_backport_labels.py
b/dev/breeze/tests/test_remove_backport_labels.py
new file mode 100644
index 00000000000..369669bab3e
--- /dev/null
+++ b/dev/breeze/tests/test_remove_backport_labels.py
@@ -0,0 +1,85 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from unittest.mock import MagicMock, patch
+
+import pytest
+
+from airflow_breeze.commands.ci_commands import remove_backport_labels
+
+MODULE = "airflow_breeze.commands.ci_commands"
+
+
+def _result(returncode: int = 0, stdout: str = "") -> MagicMock:
+ return MagicMock(returncode=returncode, stdout=stdout)
+
+
+def _call(run_command, **kwargs):
+ with patch(f"{MODULE}.run_command", run_command):
+ remove_backport_labels(branch_name="ci-upgrade-main", command_env={},
**kwargs)
+
+
+def _edit_call(run_command):
+ """Return the argv of the `gh pr edit` invocation, or None when it never
ran."""
+ for call in run_command.call_args_list:
+ argv = call.args[0]
+ if "edit" in argv:
+ return argv
+ return None
+
+
[email protected](
+ "labels",
+ ["backport-to-v3-3-test",
"backport-to-v3-3-test,backport-to-airflow-ctl/v0-1-test"],
+)
+def test_every_backport_label_is_removed_in_one_call(labels):
+ run_command = MagicMock(side_effect=[_result(stdout=labels), _result()])
+
+ _call(run_command)
+
+ argv = _edit_call(run_command)
+ assert argv is not None
+ removed = {argv[index + 1] for index, arg in enumerate(argv) if arg ==
"--remove-label"}
+ assert removed == set(labels.split(","))
+ assert run_command.call_count == 2
+
+
+def test_nothing_is_edited_when_no_backport_label_is_present():
+ run_command = MagicMock(side_effect=[_result(stdout="")])
+
+ _call(run_command)
+
+ assert _edit_call(run_command) is None
+
+
+def test_labels_are_left_alone_when_they_cannot_be_read():
+ """Without the label list we cannot tell which to remove - never guess and
edit blindly."""
+ run_command = MagicMock(side_effect=[_result(returncode=1)])
+
+ _call(run_command)
+
+ assert _edit_call(run_command) is None
+
+
+def test_a_failed_edit_does_not_raise():
+ """The upgrade PR itself is already pushed; a label left behind must not
fail the run."""
+ run_command =
MagicMock(side_effect=[_result(stdout="backport-to-v3-3-test"),
_result(returncode=1)])
+
+ _call(run_command)
+
+ assert _edit_call(run_command) is not None
diff --git a/providers/amazon/pyproject.toml b/providers/amazon/pyproject.toml
index 191329fecf2..03eaf1d6588 100644
--- a/providers/amazon/pyproject.toml
+++ b/providers/amazon/pyproject.toml
@@ -63,9 +63,9 @@ dependencies = [
"apache-airflow-providers-common-compat>=1.14.3",
"apache-airflow-providers-common-sql>=1.32.0",
"apache-airflow-providers-http",
- # We should update minimum version of boto3 and here regularly to avoid
`pip` backtracking with the number
- # of candidates to consider. Make sure to configure boto3 version here as
well as in all the tools below
- # in the `devel-dependencies` section to be the same minimum version.
+ # The boto3/botocore floors are raised automatically by `breeze ci
upgrade` to limit `pip` backtracking
+ # (see [tool.airflow.dependency-floors] in the root pyproject.toml). The
mypy-boto3-* stubs in
+ # `devel-dependencies` below are not covered by that automation and are
raised by hand.
"boto3>=1.41.0",
"botocore>=1.41.0",
"inflection>=0.5.1",
diff --git a/pyproject.toml b/pyproject.toml
index 4a836d71be3..8981569e933 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -1556,6 +1556,38 @@ apache-airflow-task-sdk-integration-tests = false
# Manual overrides (kept outside the auto-generated block above so the
# update_airflow_pyproject_toml.py script doesn't clobber them).
+[tool.airflow.dependency-floors]
+# Floors of these dependencies are raised by `breeze ci upgrade`
(scripts/ci/prek/upgrade_dependency_floors.py),
+# see dev/breeze/doc/adr/0018-raise-dependency-floors-automatically.md.
+# A floor is never raised to a release uploaded more recently than this.
+min-age = "180 days"
+packages = [
+ "boto3",
+ "botocore",
+ "google-cloud-*",
+ "google-api-python-client",
+ "azure-*",
+]
+# Members of a group always get the same floor.
+groups = [
+ ["boto3", "botocore"],
+]
+
+# Never bumped. Packages capped or excluded anywhere (<, <=, !=, ==, ~=) are
skipped automatically;
+# list here the ones that must stay put for another reason, or whose cap
should not be the only record.
+[tool.airflow.dependency-floors.exclude]
+sagemaker-studio = "Do not change without approval from AWS
(providers/amazon/pyproject.toml)"
+pandas = "DataFrame XComs need pandas<3 (#70791)"
+pymysql = "Capped below 1.2 in PyPI constraints generation (#67491)"
+fastapi = "Repeated breakage on new releases (#59681, #59710, #59856, #61579,
#68578)"
+deltalake = "Missing ARM wheels on new releases (#59977, #60098, #60376)"
+gunicorn = "API server startup deadlocks above 25.1 (#62524)"
+qdrant-client = "Capped (#62193)"
+airbyte-api = "1.x breaks provider tests (#69081)"
+mysql-connector-python = "Releases without wheels for >=3.12 (#60889, #66026)"
+anthropic = "Provider SDK migration is version-sensitive (#72072, #72094)"
+azure-ai-projects = "2.5+ needs openai>=3, broke main (#73621, #73627)"
+
[tool.uv.pip]
# Synchroonize with scripts/ci/prek/upgrade_important_versions.py
exclude-newer = "4 days"
diff --git a/scripts/ci/prek/upgrade_dependency_floors.py
b/scripts/ci/prek/upgrade_dependency_floors.py
new file mode 100755
index 00000000000..3813e3711ef
--- /dev/null
+++ b/scripts/ci/prek/upgrade_dependency_floors.py
@@ -0,0 +1,537 @@
+#!/usr/bin/env python
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# /// script
+# requires-python = ">=3.10,<3.11"
+# dependencies = [
+# "packaging>=25",
+# "requests>=2.31.0",
+# "rich>=13.6.0",
+# "tomli>=2.0.1",
+# ]
+# ///
+"""
+Raise the lower bounds of a curated set of dependencies to the newest release
older than a cooldown.
+
+Policy lives in ``[tool.airflow.dependency-floors]`` of the root
``pyproject.toml``; see
+dev/breeze/doc/adr/0018-raise-dependency-floors-automatically.md.
+"""
+
+from __future__ import annotations
+
+import os
+import re
+import subprocess
+import sys
+from collections import defaultdict
+from collections.abc import Callable, Iterable
+from dataclasses import dataclass
+from datetime import datetime, timedelta, timezone
+from fnmatch import fnmatchcase
+from pathlib import Path
+
+import requests
+from check_dependency_lower_bounds import extract_requirements,
get_workspace_distribution_names
+from common_prek_utils import AIRFLOW_ROOT_PATH, console
+from packaging.requirements import InvalidRequirement, Requirement
+from packaging.utils import canonicalize_name
+from packaging.version import InvalidVersion, Version
+
+try:
+ import tomllib
+except ImportError:
+ import tomli as tomllib # type: ignore[no-redef]
+
+_DURATION_RE = re.compile(r"^\s*(\d+(?:\.\d+)?)\s*(minute|hour|day)s?\s*$")
+
+
+@dataclass(frozen=True)
+class FloorConfig:
+ min_age: timedelta
+ packages: tuple[str, ...]
+ groups: tuple[tuple[str, ...], ...]
+ exclude: dict[str, str]
+
+
+def parse_duration(value: str) -> timedelta:
+ """Parse the ``N days|hours|minutes`` form uv accepts in
``exclude-newer-package``."""
+ match = _DURATION_RE.match(value)
+ if not match:
+ raise ValueError(f"Invalid duration {value!r}: expected e.g. '180
days', '12 hours'")
+ return timedelta(**{f"{match.group(2)}s": float(match.group(1))})
+
+
+def _canonicalize_pattern(pattern: str) -> str:
+ # canonicalize_name keeps "*" and "?" intact, so globs normalize like
names do.
+ return canonicalize_name(pattern)
+
+
+def is_curated(name: str, config: FloorConfig) -> bool:
+ canonical = canonicalize_name(name)
+ return any(fnmatchcase(canonical, pattern) for pattern in config.packages)
+
+
+def load_config(pyproject_path: Path) -> FloorConfig:
+ data = tomllib.loads(pyproject_path.read_text())
+ section = data.get("tool", {}).get("airflow", {}).get("dependency-floors")
+ if section is None:
+ raise ValueError(f"No [tool.airflow.dependency-floors] section in
{pyproject_path}")
+ config = FloorConfig(
+ min_age=parse_duration(section["min-age"]),
+ packages=tuple(_canonicalize_pattern(p) for p in
section.get("packages", [])),
+ groups=tuple(tuple(canonicalize_name(m) for m in group) for group in
section.get("groups", [])),
+ exclude={canonicalize_name(k): v for k, v in section.get("exclude",
{}).items()},
+ )
+ for group in config.groups:
+ for member in group:
+ if not is_curated(member, config):
+ raise ValueError(f"Group member {member!r} is not covered by
'packages'")
+ return config
+
+
+# A site with any of these means we are deliberately holding the package back.
+HOLD_BACK_OPERATORS = {"<", "<=", "!=", "==", "~=", "==="}
+
+
+@dataclass(frozen=True)
+class RequirementSite:
+ path: Path
+ section: str
+ raw: str
+ requirement: Requirement
+
+
+def find_requirements(
+ pyproject_paths: Iterable[Path], workspace_names: frozenset[str]
+) -> dict[str, list[RequirementSite]]:
+ sites: dict[str, list[RequirementSite]] = defaultdict(list)
+ for path in pyproject_paths:
+ for section, raw in
extract_requirements(tomllib.loads(path.read_text())):
+ if section == "build-system.requires":
+ continue
+ try:
+ requirement = Requirement(raw)
+ except InvalidRequirement:
+ # check-dependency-lower-bounds reports these
+ continue
+ name = canonicalize_name(requirement.name)
+ if requirement.url or name in workspace_names:
+ continue
+ sites[name].append(RequirementSite(path=path, section=section,
raw=raw, requirement=requirement))
+ return dict(sites)
+
+
+def _get_display_path(path: Path) -> Path:
+ # Reasons end up in the upgrade PR description, so do not leak the local
checkout location.
+ try:
+ return path.relative_to(AIRFLOW_ROOT_PATH)
+ except ValueError:
+ return path
+
+
+UV_HOLD_TABLES = ("constraint-dependencies", "override-dependencies")
+
+
+def find_uv_holds(root_pyproject: Path) -> dict[str, list[RequirementSite]]:
+ """Requirements in the root ``[tool.uv]`` tables that cap packages without
declaring a dependency."""
+ tool_uv = tomllib.loads(root_pyproject.read_text()).get("tool",
{}).get("uv", {})
+ holds: dict[str, list[RequirementSite]] = defaultdict(list)
+ for table in UV_HOLD_TABLES:
+ for raw in tool_uv.get(table, []):
+ requirement = Requirement(raw)
+ holds[canonicalize_name(requirement.name)].append(
+ RequirementSite(
+ path=root_pyproject, section=f"tool.uv.{table}", raw=raw,
requirement=requirement
+ )
+ )
+ return dict(holds)
+
+
+def _get_site_location(site: RequirementSite) -> str:
+ if site.section.startswith("tool.uv."):
+ return f"[tool.uv] {site.section.removeprefix('tool.uv.')}"
+ return str(_get_display_path(site.path))
+
+
+def get_exclusion_reason(name: str, sites: list[RequirementSite], config:
FloorConfig) -> str | None:
+ if reason := config.exclude.get(canonicalize_name(name)):
+ return reason
+ for site in sites:
+ held = [str(s) for s in site.requirement.specifier if s.operator in
HOLD_BACK_OPERATORS]
+ if held:
+ return f"held back by {','.join(held)} in
{_get_site_location(site)}"
+ return None
+
+
+def _parse_upload_time(value: str) -> datetime:
+ return datetime.fromisoformat(value.replace("Z", "+00:00"))
+
+
+def get_eligible_versions(releases: dict[str, list[dict]], min_age: timedelta,
now: datetime) -> set[Version]:
+ cutoff = now - min_age
+ eligible: set[Version] = set()
+ for version_str, files in releases.items():
+ if not files or any(f.get("yanked") for f in files):
+ continue
+ try:
+ version = Version(version_str)
+ except InvalidVersion:
+ continue
+ if version.is_prerelease or version.is_devrelease:
+ continue
+ if min(_parse_upload_time(f["upload_time_iso_8601"]) for f in files)
<= cutoff:
+ eligible.add(version)
+ return eligible
+
+
+def find_target_version(releases: dict[str, list[dict]], min_age: timedelta,
now: datetime) -> Version | None:
+ return max(get_eligible_versions(releases, min_age, now), default=None)
+
+
+def find_group_target(
+ releases_by_member: dict[str, dict[str, list[dict]]], min_age: timedelta,
now: datetime
+) -> Version | None:
+ eligible = [get_eligible_versions(r, min_age, now) for r in
releases_by_member.values()]
+ return max(set.intersection(*eligible), default=None) if eligible else None
+
+
+FLOOR_OPERATORS = {">=", ">"}
+
+
+def get_floor(requirement: Requirement) -> Version | None:
+ floors = [Version(s.version) for s in requirement.specifier if s.operator
in FLOOR_OPERATORS]
+ return max(floors, default=None)
+
+
+def rewrite_requirement(raw: str, target: Version) -> str | None:
+ """Return ``raw`` with its floor raised to ``target``, or None when it
would not be raised."""
+ requirement = Requirement(raw)
+ floor = get_floor(requirement)
+ if floor is None or target <= floor:
+ return None
+ rewritten = raw
+ for specifier in requirement.specifier:
+ if specifier.operator in FLOOR_OPERATORS and
Version(specifier.version) == floor:
+ pattern = re.compile(
+
rf"{re.escape(specifier.operator)}(\s*){re.escape(specifier.version)}(?![\w.])"
+ )
+ rewritten = pattern.sub(lambda m: f">={m.group(1)}{target}",
rewritten, count=1)
+ return rewritten
+
+
+@dataclass(frozen=True)
+class Edit:
+ path: Path
+ old: str
+ new: str
+
+
+@dataclass(frozen=True)
+class Bump:
+ unit: str
+ packages: tuple[str, ...]
+ old_floors: tuple[str, ...]
+ target: Version
+ edits: tuple[Edit, ...]
+
+
+def build_bump(unit: str, sites: list[RequirementSite], target: Version) ->
Bump | None:
+ edits: dict[tuple[Path, str], Edit] = {}
+ old_floors: set[str] = set()
+ packages: set[str] = set()
+ for site in sites:
+ if (new := rewrite_requirement(site.raw, target)) is not None:
+ edits[(site.path, site.raw)] = Edit(path=site.path, old=site.raw,
new=new)
+ old_floors.add(str(get_floor(site.requirement)))
+ packages.add(canonicalize_name(site.requirement.name))
+ if not edits:
+ return None
+ return Bump(
+ unit=unit,
+ packages=tuple(sorted(packages)),
+ old_floors=tuple(sorted(old_floors, key=Version)),
+ target=target,
+ edits=tuple(edits.values()),
+ )
+
+
+# Which quoted occurrences of ``edit.new`` each applied edit produced, so a
revert restores only
+# those and never lowers an identical requirement that was already at the
target.
+_APPLIED_OCCURRENCES: dict[Edit, list[int]] = {}
+
+
+def _find_quoted(content: str, text: str) -> list[int]:
+ """Return the start offsets of ``text`` quoted with either TOML quote
character."""
+ return sorted(
+ match.start()
+ for quote in ('"', "'")
+ for match in re.finditer(re.escape(f"{quote}{text}{quote}"), content)
+ )
+
+
+def _replace_at(content: str, starts: list[int], old: str, new: str) -> str:
+ for start in sorted(starts, reverse=True):
+ quote = content[start]
+ content = content[:start] + f"{quote}{new}{quote}" + content[start +
len(old) + 2 :]
+ return content
+
+
+def apply_bump(bump: Bump) -> None:
+ for edit in bump.edits:
+ if not _find_quoted(edit.path.read_text(), edit.old):
+ raise ValueError(f"Requirement {edit.old!r} not found in
{edit.path}")
+ for edit in bump.edits:
+ content = edit.path.read_text()
+ old_starts = _find_quoted(content, edit.old)
+ updated = _replace_at(content, old_starts, edit.old, edit.new)
+ shift = len(edit.new) - len(edit.old)
+ new_starts = {start + shift * index for index, start in
enumerate(old_starts)}
+ _APPLIED_OCCURRENCES[edit] = [
+ index for index, start in enumerate(_find_quoted(updated,
edit.new)) if start in new_starts
+ ]
+ edit.path.write_text(updated)
+
+
+def revert_bump(bump: Bump) -> None:
+ for edit in reversed(bump.edits):
+ content = edit.path.read_text()
+ occurrences = _find_quoted(content, edit.new)
+ starts = [occurrences[index] for index in
_APPLIED_OCCURRENCES.pop(edit)]
+ edit.path.write_text(_replace_at(content, starts, edit.new, edit.old))
+
+
+RESOLVE_COMMANDS: tuple[tuple[str, ...], ...] = (
+ ("uv", "lock", "--dry-run"),
+ ("uv", "lock", "--dry-run", "--resolution", "lowest-direct"),
+)
+ERROR_TAIL_LINES = 20
+
+
+@dataclass(frozen=True)
+class ResolveResult:
+ ok: bool
+ error: str = ""
+
+
+class LockAlreadyBrokenError(Exception):
+ """The workspace does not resolve even without any floor bump."""
+
+
+def resolve_check(root: Path) -> ResolveResult:
+ for command in RESOLVE_COMMANDS:
+ result = subprocess.run(list(command), cwd=root, capture_output=True,
text=True, check=False)
+ if result.returncode != 0:
+ tail =
"\n".join(result.stderr.strip().splitlines()[-ERROR_TAIL_LINES:])
+ return ResolveResult(ok=False, error=f"$ {'
'.join(command)}\n{tail}")
+ return ResolveResult(ok=True)
+
+
+def _check_with(bumps: list[Bump], check: Callable[[], ResolveResult]) ->
ResolveResult:
+ applied: list[Bump] = []
+ try:
+ for bump in bumps:
+ apply_bump(bump)
+ applied.append(bump)
+ return check()
+ finally:
+ for bump in reversed(applied):
+ revert_bump(bump)
+
+
+def _find_failing(
+ bumps: list[Bump], check: Callable[[], ResolveResult], context: list[Bump]
+) -> list[tuple[Bump, str]]:
+ """Return the bumps that break resolution when applied on top of
``context`` (known-good bumps)."""
+ result = _check_with(context + bumps, check)
+ if result.ok:
+ return []
+ if len(bumps) == 1:
+ return [(bumps[0], result.error)]
+ middle = len(bumps) // 2
+ left, right = bumps[:middle], bumps[middle:]
+ failing_left = _find_failing(left, check, context)
+ failing_units = {bump.unit for bump, _ in failing_left}
+ # The right half is judged on top of the surviving left half, so a failure
that needs
+ # one bump from each half is pinned on the right-hand one instead of on
both halves.
+ surviving_left = [bump for bump in left if bump.unit not in failing_units]
+ return failing_left + _find_failing(right, check, context + surviving_left)
+
+
+def apply_with_rollback(
+ bumps: list[Bump], check: Callable[[], ResolveResult]
+) -> tuple[list[Bump], list[tuple[Bump, str]]]:
+ rolled_back: list[tuple[Bump, str]] = []
+ remaining = list(bumps)
+ while remaining:
+ if _check_with(remaining, check).ok:
+ break
+ if not rolled_back and not (baseline := check()).ok:
+ raise LockAlreadyBrokenError(baseline.error)
+ failing = _find_failing(remaining, check, context=[])
+ failing_units = {bump.unit for bump, _ in failing}
+ rolled_back.extend(failing)
+ remaining = [bump for bump in remaining if bump.unit not in
failing_units]
+ for bump in remaining:
+ apply_bump(bump)
+ return remaining, rolled_back
+
+
+PYPI_TIMEOUT_SECONDS = 30
+# Without a cut-off an unreachable PyPI would cost PYPI_TIMEOUT_SECONDS for
every curated package.
+MAX_CONSECUTIVE_CONNECTION_FAILURES = 3
+REPORT_ENV = "DEPENDENCY_FLOORS_REPORT"
+
+
+@dataclass
+class Report:
+ raised: list[Bump]
+ skipped: dict[str, str]
+ rolled_back: list[tuple[Bump, str]]
+
+
+def fetch_releases(name: str) -> dict[str, list[dict]]:
+ response = requests.get(f"https://pypi.org/pypi/{name}/json";,
timeout=PYPI_TIMEOUT_SECONDS)
+ response.raise_for_status()
+ return response.json()["releases"]
+
+
+def _get_units(names: list[str], config: FloorConfig) -> list[tuple[str,
tuple[str, ...]]]:
+ """Group curated names into bump units; a group is a unit only when a
member is present."""
+ units: list[tuple[str, tuple[str, ...]]] = []
+ grouped: set[str] = set()
+ for group in config.groups:
+ if any(member in names for member in group):
+ units.append(("+".join(group), group))
+ grouped.update(group)
+ units.extend((name, (name,)) for name in names if name not in grouped)
+ return units
+
+
+def run(
+ root: Path,
+ pyproject_paths: list[Path],
+ workspace_names: frozenset[str],
+ now: datetime,
+ fetch: Callable[[str], dict[str, list[dict]]],
+ check: Callable[[], ResolveResult],
+) -> Report:
+ config = load_config(root / "pyproject.toml")
+ sites = find_requirements(pyproject_paths, workspace_names)
+ uv_holds = find_uv_holds(root / "pyproject.toml")
+ curated = sorted(name for name in sites if is_curated(name, config))
+ skipped: dict[str, str] = {}
+ bumps: list[Bump] = []
+ connection_failures = 0
+ for unit, members in _get_units(curated, config):
+ member_sites = [site for member in members for site in
sites.get(member, [])]
+ reasons = (
+ get_exclusion_reason(member, sites.get(member, []) +
uv_holds.get(member, []), config)
+ for member in members
+ )
+ if reason := next((r for r in reasons if r), None):
+ skipped[unit] = reason
+ continue
+ if connection_failures >= MAX_CONSECUTIVE_CONNECTION_FAILURES:
+ skipped[unit] = f"PyPI unreachable ({connection_failures}
consecutive connection failures)"
+ continue
+ try:
+ releases = {member: fetch(member) for member in members}
+ target = find_group_target(releases, config.min_age, now)
+ except (requests.HTTPError, KeyError, TypeError, ValueError) as error:
+ # PyPI answered, just not with usable data for this package.
+ skipped[unit] = f"PyPI metadata unavailable: {error!r}"
+ continue
+ except OSError as error:
+ # requests' own exceptions subclass OSError, so this covers
connection errors and timeouts.
+ connection_failures += 1
+ skipped[unit] = f"PyPI metadata unavailable: {error!r}"
+ continue
+ connection_failures = 0
+ if target is None:
+ skipped[unit] = f"no release older than {config.min_age.days} days"
+ continue
+ if bump := build_bump(unit, member_sites, target):
+ bumps.append(bump)
+ raised, rolled_back = apply_with_rollback(bumps, check)
+ return Report(raised=raised, skipped=skipped, rolled_back=rolled_back)
+
+
+def render_report(report: Report) -> str:
+ lines = ["### Dependency floors", ""]
+ lines.append("Raised:" if report.raised else "Raised: none")
+ for bump in report.raised:
+ names = ", ".join(f"`{p}`" for p in bump.packages)
+ files = ", ".join(sorted({str(_get_display_path(edit.path)) for edit
in bump.edits}))
+ lines.append(f"- {names}: {', '.join(bump.old_floors)} → {bump.target}
({files})")
+ if report.rolled_back:
+ lines += ["", "Rolled back (resolution failed):"]
+ for bump, error in report.rolled_back:
+ lines.append(f"- `{bump.unit}` → {bump.target}")
+ lines += [" ```", *(f" {line}" for line in error.splitlines()),
" ```"]
+ if report.skipped:
+ lines += ["", "Skipped:"]
+ lines += [f"- `{unit}`: {reason}" for unit, reason in
sorted(report.skipped.items())]
+ return "\n".join(lines) + "\n"
+
+
+def get_workspace_pyprojects(root: Path) -> list[Path]:
+ """Return the root and member pyproject.toml files whose floors the root
``uv.lock`` validates.
+
+ Members with a ``uv.lock`` of their own (dev/breeze) are left out: the
resolve check only
+ covers the root lock, so raising their floors would leave their lock stale.
+ """
+ data = tomllib.loads((root / "pyproject.toml").read_text())
+ members = data.get("tool", {}).get("uv", {}).get("workspace",
{}).get("members", [])
+ member_pyprojects = sorted(
+ path
+ for member in members
+ for path in root.glob(f"{member}/pyproject.toml")
+ if path.parent != root and not (path.parent / "uv.lock").exists()
+ )
+ return list(dict.fromkeys([root / "pyproject.toml", *member_pyprojects]))
+
+
+def main() -> int:
+ root = AIRFLOW_ROOT_PATH
+ try:
+ report = run(
+ root,
+ get_workspace_pyprojects(root),
+ get_workspace_distribution_names(),
+ datetime.now(timezone.utc),
+ fetch_releases,
+ lambda: resolve_check(root),
+ )
+ except (ValueError, LockAlreadyBrokenError) as error:
+ console.print(f"[red]Dependency floors not updated: {error}[/]")
+ # Say so in the upgrade PR too, or a broken step would go unnoticed
run after run.
+ _write_report(f"### Dependency floors\n\nNot updated: {error}\n")
+ return 1
+ text = render_report(report)
+ console.print(text)
+ _write_report(text)
+ return 0
+
+
+def _write_report(text: str) -> None:
+ if report_path := os.environ.get(REPORT_ENV):
+ Path(report_path).write_text(text)
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/scripts/tests/ci/prek/test_upgrade_dependency_floors.py
b/scripts/tests/ci/prek/test_upgrade_dependency_floors.py
new file mode 100644
index 00000000000..5454f723537
--- /dev/null
+++ b/scripts/tests/ci/prek/test_upgrade_dependency_floors.py
@@ -0,0 +1,667 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import subprocess
+import textwrap
+from datetime import datetime, timedelta, timezone
+from pathlib import Path
+from unittest import mock
+
+import pytest
+import requests
+from common_prek_utils import AIRFLOW_ROOT_PATH
+from packaging.requirements import Requirement
+from packaging.version import Version
+from upgrade_dependency_floors import (
+ RESOLVE_COMMANDS,
+ Bump,
+ Edit,
+ FloorConfig,
+ LockAlreadyBrokenError,
+ Report,
+ RequirementSite,
+ ResolveResult,
+ apply_bump,
+ apply_with_rollback,
+ build_bump,
+ find_group_target,
+ find_requirements,
+ find_target_version,
+ get_exclusion_reason,
+ get_workspace_pyprojects,
+ is_curated,
+ load_config,
+ main,
+ parse_duration,
+ render_report,
+ resolve_check,
+ revert_bump,
+ rewrite_requirement,
+ run,
+)
+
+CONFIG = """
+[tool.airflow.dependency-floors]
+min-age = "180 days"
+packages = ["boto3", "botocore", "Google_Cloud-*"]
+groups = [["boto3", "botocore"]]
+
+[tool.airflow.dependency-floors.exclude]
+Sagemaker_Studio = "Ask AWS first"
+"""
+
+
+def _write(tmp_path, content):
+ path = tmp_path / "pyproject.toml"
+ path.write_text(textwrap.dedent(content))
+ return path
+
+
[email protected](
+ ("value", "expected"),
+ [
+ pytest.param("180 days", timedelta(days=180), id="days"),
+ pytest.param("1 day", timedelta(days=1), id="singular"),
+ pytest.param("12 hours", timedelta(hours=12), id="hours"),
+ pytest.param("30 minutes", timedelta(minutes=30), id="minutes"),
+ pytest.param(" 1.5 days ", timedelta(days=1.5),
id="fraction-and-spaces"),
+ ],
+)
+def test_parse_duration(value, expected):
+ assert parse_duration(value) == expected
+
+
[email protected]("value", ["", "6 months", "days", "-1 days"])
+def test_parse_duration_rejects(value):
+ with pytest.raises(ValueError, match="duration"):
+ parse_duration(value)
+
+
+def test_load_config_canonicalizes_patterns(tmp_path):
+ config = load_config(_write(tmp_path, CONFIG))
+ assert config == FloorConfig(
+ min_age=timedelta(days=180),
+ packages=("boto3", "botocore", "google-cloud-*"),
+ groups=(("boto3", "botocore"),),
+ exclude={"sagemaker-studio": "Ask AWS first"},
+ )
+
+
+def test_load_config_missing_section(tmp_path):
+ with pytest.raises(ValueError,
match=r"\[tool.airflow.dependency-floors\]"):
+ load_config(_write(tmp_path, "[project]\nname = 'x'\n"))
+
+
+def test_load_config_group_member_not_curated(tmp_path):
+ content = CONFIG.replace('groups = [["boto3", "botocore"]]', 'groups =
[["boto3", "aiobotocore"]]')
+ with pytest.raises(ValueError, match="aiobotocore"):
+ load_config(_write(tmp_path, content))
+
+
[email protected](
+ ("name", "expected"),
+ [
+ pytest.param("boto3", True, id="exact"),
+ pytest.param("google-cloud-storage", True, id="glob"),
+ pytest.param("google_cloud_storage", True, id="non-canonical"),
+ pytest.param("google-api-core", False, id="no-match"),
+ ],
+)
+def test_is_curated(tmp_path, name, expected):
+ assert is_curated(name, load_config(_write(tmp_path, CONFIG))) is expected
+
+
+def test_repository_config_loads():
+ config = load_config(AIRFLOW_ROOT_PATH / "pyproject.toml")
+ assert config.min_age == timedelta(days=180)
+ assert ("boto3", "botocore") in config.groups
+
+
+PROVIDER = """
+[project]
+name = "apache-airflow-providers-amazon"
+dependencies = [
+ "Boto3>=1.41.0",
+ "apache-airflow-core>=3.0.0",
+ "foo @ https://example.com/foo.whl";,
+]
+[project.optional-dependencies]
+"s3fs" = ["s3fs>=2023.10.0"]
+[dependency-groups]
+dev = ["boto3>=1.41.0", {include-group = "docs"}]
+[build-system]
+requires = ["hatchling==1.31.0"]
+"""
+
+
+def test_find_requirements_canonicalizes_names(tmp_path):
+ path = _write(tmp_path, PROVIDER)
+ sites = find_requirements([path], frozenset({"apache-airflow-core"}))
+ assert [(s.section, s.raw) for s in sites["boto3"]] == [
+ ("project.dependencies", "Boto3>=1.41.0"),
+ ('dependency-groups."dev"', "boto3>=1.41.0"),
+ ]
+
+
+def test_find_requirements_skips_workspace_and_urls(tmp_path):
+ sites = find_requirements([_write(tmp_path, PROVIDER)],
frozenset({"apache-airflow-core"}))
+ assert "apache-airflow-core" not in sites
+ assert "foo" not in sites
+ # build-system.requires is never edited
+ assert "hatchling" not in sites
+
+
+def _site(raw: str, path: str = "p/pyproject.toml") -> RequirementSite:
+ return RequirementSite(
+ path=Path(path), section="project.dependencies", raw=raw,
requirement=Requirement(raw)
+ )
+
+
[email protected](
+ "raw",
+ [
+ pytest.param("boto3>=1.41,<2", id="upper"),
+ pytest.param("boto3>=1.41,<=1.50", id="upper-inclusive"),
+ pytest.param("boto3>=1.41,!=1.42.0", id="exclusion"),
+ pytest.param("boto3==1.41.0", id="pin"),
+ pytest.param("boto3~=1.41", id="compatible"),
+ pytest.param("boto3===1.41.0", id="arbitrary"),
+ ],
+)
+def test_exclusion_when_held_back(tmp_path, raw):
+ config = load_config(_write(tmp_path, CONFIG))
+ reason = get_exclusion_reason("boto3", [_site("boto3>=1.40"), _site(raw,
"q/pyproject.toml")], config)
+ assert reason is not None
+ assert "q/pyproject.toml" in reason
+
+
+def test_exclusion_explicit_list(tmp_path):
+ config = load_config(_write(tmp_path, CONFIG))
+ assert (
+ get_exclusion_reason("sagemaker-studio",
[_site("sagemaker-studio>=1.0")], config) == "Ask AWS first"
+ )
+
+
+def test_no_exclusion_for_plain_floor(tmp_path):
+ config = load_config(_write(tmp_path, CONFIG))
+ assert get_exclusion_reason("boto3", [_site("boto3>=1.40; python_version <
'3.14'")], config) is None
+
+
+NOW = datetime(2026, 9, 24, tzinfo=timezone.utc)
+AGE = timedelta(days=180)
+
+
+def _files(*uploads: str, yanked: bool = False) -> list[dict]:
+ return [{"upload_time_iso_8601": u, "yanked": yanked} for u in uploads]
+
+
+RELEASES = {
+ "1.0.0": _files("2025-01-01T00:00:00.000000Z"),
+ # the earliest upload of a release counts
+ "1.1.0": _files("2026-03-01T00:00:00.000000Z",
"2026-06-01T00:00:00.000000Z"),
+ "1.2.0": _files("2026-03-10T00:00:00.000000Z", yanked=True),
+ "1.3.0rc1": _files("2026-01-01T00:00:00.000000Z"),
+ "1.3.0.dev1": _files("2026-01-01T00:00:00.000000Z"),
+ "1.4.0": _files("2026-09-01T00:00:00.000000Z"),
+ "1.5.0": [],
+}
+
+
+def test_target_is_newest_old_enough_final_release():
+ assert find_target_version(RELEASES, AGE, NOW) == Version("1.1.0")
+
+
[email protected](
+ ("upload", "eligible"),
+ [
+ pytest.param("2026-03-28T00:00:00.000000Z", True,
id="exactly-min-age"),
+ pytest.param("2026-03-28T00:00:01.000000Z", False,
id="one-second-too-new"),
+ ],
+)
+def test_min_age_boundary(upload, eligible):
+ target = find_target_version({"2.0.0": _files(upload)}, AGE, NOW)
+ assert (target == Version("2.0.0")) is eligible
+
+
+def test_no_release_old_enough():
+ assert find_target_version({"9.0.0":
_files("2026-09-20T00:00:00.000000Z")}, AGE, NOW) is None
+
+
+def test_group_target_is_common_version():
+ boto3 = {"1.40.0": _files("2026-01-01T00:00:00Z"), "1.40.5":
_files("2026-02-01T00:00:00Z")}
+ botocore = {"1.40.0": _files("2026-01-01T00:00:00Z"), "1.40.3":
_files("2026-02-01T00:00:00Z")}
+ assert find_group_target({"boto3": boto3, "botocore": botocore}, AGE, NOW)
== Version("1.40.0")
+
+
+def test_group_without_common_version():
+ boto3 = {"1.40.5": _files("2026-01-01T00:00:00Z")}
+ botocore = {"1.40.3": _files("2026-01-01T00:00:00Z")}
+ assert find_group_target({"boto3": boto3, "botocore": botocore}, AGE, NOW)
is None
+
+
[email protected](
+ ("raw", "expected"),
+ [
+ pytest.param("boto3>=1.41.0", "boto3>=1.43.0", id="plain"),
+ pytest.param("boto3 >= 1.41.0", "boto3 >= 1.43.0", id="spaced"),
+ pytest.param("boto3>1.41.0", "boto3>=1.43.0", id="exclusive"),
+ pytest.param("pydantic-ai-slim[mcp]>=1.0",
"pydantic-ai-slim[mcp]>=1.43.0", id="extras"),
+ pytest.param(
+ "boto3>=1.41.0; python_version < '3.14'", "boto3>=1.43.0;
python_version < '3.14'", id="marker"
+ ),
+ pytest.param("boto3>=1.44.0; python_version >= '3.14'", None,
id="floor-already-higher"),
+ pytest.param("boto3>=1.43.0", None, id="floor-equal"),
+ pytest.param("boto3", None, id="no-floor"),
+ ],
+)
+def test_rewrite(raw, expected):
+ assert rewrite_requirement(raw, Version("1.43.0")) == expected
+
+
+def test_build_bump_skips_sites_already_high():
+ low, high = _site("boto3>=1.41.0"), _site("boto3>=1.44.0; python_version
>= '3.14'")
+ bump = build_bump("boto3", [low, high], Version("1.43.0"))
+ assert bump.edits == (Edit(path=low.path, old="boto3>=1.41.0",
new="boto3>=1.43.0"),)
+ assert bump.old_floors == ("1.41.0",)
+
+
+def test_build_bump_nothing_to_raise():
+ assert build_bump("boto3", [_site("boto3>=1.44.0")], Version("1.43.0")) is
None
+
+
[email protected](
+ "quote",
+ [pytest.param('"', id="apply_bump_double_quotes"), pytest.param("'",
id="apply_bump_single_quotes")],
+)
+def test_apply_and_revert_bump(tmp_path, quote):
+ path = tmp_path / "pyproject.toml"
+ original = f"dependencies = [\n {quote}boto3>=1.41.0{quote}, # keep
me\n]\n"
+ path.write_text(original)
+ bump = build_bump("boto3", [_site("boto3>=1.41.0", str(path))],
Version("1.43.0"))
+ apply_bump(bump)
+ assert path.read_text() == original.replace("1.41.0", "1.43.0")
+ revert_bump(bump)
+ assert path.read_text() == original
+
+
+def test_apply_bump_missing_text_raises(tmp_path):
+ path = tmp_path / "pyproject.toml"
+ path.write_text("dependencies = []\n")
+ bump = build_bump("boto3", [_site("boto3>=1.41.0", str(path))],
Version("1.43.0"))
+ with pytest.raises(ValueError, match="boto3>=1.41.0"):
+ apply_bump(bump)
+
+
+def _bump(unit: str) -> Bump:
+ return Bump(unit=unit, packages=(unit,), old_floors=("1.0",),
target=Version("2.0"), edits=())
+
+
+class FakeWorkspace:
+ """Tracks which bumps are applied; the check fails while any 'bad' set is
fully applied."""
+
+ def __init__(self, bad_sets: list[set[str]], broken: bool = False):
+ self.applied: set[str] = set()
+ self.bad_sets = bad_sets
+ self.broken = broken
+
+ def apply(self, bump):
+ self.applied.add(bump.unit)
+
+ def revert(self, bump):
+ self.applied.discard(bump.unit)
+
+ def check(self) -> ResolveResult:
+ if self.broken:
+ return ResolveResult(ok=False, error="lock broken")
+ for bad in self.bad_sets:
+ if bad <= self.applied:
+ return ResolveResult(ok=False, error=f"conflict {sorted(bad)}")
+ return ResolveResult(ok=True)
+
+
[email protected]
+def workspace(monkeypatch):
+ def _make(bad_sets, broken=False):
+ ws = FakeWorkspace(bad_sets, broken)
+ monkeypatch.setattr("upgrade_dependency_floors.apply_bump", ws.apply)
+ monkeypatch.setattr("upgrade_dependency_floors.revert_bump", ws.revert)
+ return ws
+
+ return _make
+
+
+def test_rollback_nothing_when_green(workspace):
+ ws = workspace([])
+ applied, rolled_back = apply_with_rollback([_bump(u) for u in "abcd"],
ws.check)
+ assert [b.unit for b in applied] == list("abcd")
+ assert rolled_back == []
+ assert ws.applied == set("abcd")
+
+
+def test_rollback_single_bad(workspace):
+ ws = workspace([{"c"}])
+ applied, rolled_back = apply_with_rollback([_bump(u) for u in "abcdefg"],
ws.check)
+ assert [(b.unit, e) for b, e in rolled_back] == [("c", "conflict ['c']")]
+ assert ws.applied == set("abdefg")
+
+
+def test_rollback_interacting_pair(workspace):
+ ws = workspace([{"b", "e"}])
+ applied, rolled_back = apply_with_rollback([_bump(u) for u in "abcdef"],
ws.check)
+ assert ws.check().ok
+ # Only one side of the conflicting pair is dropped, not the whole batch.
+ assert [b.unit for b, _ in rolled_back] == ["e"]
+ assert ws.applied == set("abcdf") == {b.unit for b in applied}
+
+
+def test_lock_broken_before_bumps(workspace):
+ ws = workspace([], broken=True)
+ with pytest.raises(LockAlreadyBrokenError, match="lock broken"):
+ apply_with_rollback([_bump("a")], ws.check)
+ assert ws.applied == set()
+
+
[email protected]("upgrade_dependency_floors.subprocess.run", autospec=True)
+def test_resolve_check_runs_both_resolutions(mock_run, tmp_path):
+ mock_run.return_value = mock.Mock(spec=subprocess.CompletedProcess,
returncode=0, stderr="")
+ assert resolve_check(tmp_path) == ResolveResult(ok=True)
+ assert [c.args[0] for c in mock_run.call_args_list] == [list(cmd) for cmd
in RESOLVE_COMMANDS]
+
+
[email protected]("upgrade_dependency_floors.subprocess.run", autospec=True)
+def test_resolve_check_reports_stderr_tail(mock_run, tmp_path):
+ mock_run.return_value = mock.Mock(
+ spec=subprocess.CompletedProcess, returncode=1,
stderr="\n".join(f"line {i}" for i in range(50))
+ )
+ result = resolve_check(tmp_path)
+ assert not result.ok
+ assert result.error.splitlines()[-1] == "line 49"
+ assert "line 0" not in result.error
+
+
+ROOT_CONFIG = CONFIG + '\n[tool.uv.workspace]\nmembers =
["providers/amazon"]\n'
+AMAZON = """
+[project]
+name = "apache-airflow-providers-amazon"
+dependencies = [
+ "boto3>=1.41.0",
+ "botocore>=1.41.0",
+ "google-cloud-storage>=2.0.0,<3",
+ "google-cloud-bigquery>=3.0.0",
+ "sagemaker-studio>=1.0.25",
+ "apache-airflow-providers-google>=1.0",
+]
+"""
+OLD = "2026-01-01T00:00:00Z"
+PYPI = {
+ "boto3": {"1.41.0": _files(OLD), "1.42.0": _files(OLD), "1.50.0":
_files("2026-09-01T00:00:00Z")},
+ "botocore": {"1.41.0": _files(OLD), "1.42.0": _files(OLD)},
+ "google-cloud-bigquery": {"3.0.0": _files(OLD), "3.9.0": _files(OLD)},
+}
+WORKSPACE = frozenset({"apache-airflow-providers-google"})
+
+
[email protected]
+def tree(tmp_path):
+ (tmp_path / "pyproject.toml").write_text(ROOT_CONFIG)
+ (tmp_path / "providers/amazon").mkdir(parents=True)
+ amazon = tmp_path / "providers/amazon/pyproject.toml"
+ amazon.write_text(AMAZON)
+ return tmp_path, amazon
+
+
+def _fetch(name):
+ if name not in PYPI:
+ raise OSError(f"{name} not on PyPI")
+ return PYPI[name]
+
+
+def _green():
+ return ResolveResult(ok=True)
+
+
+def test_run_raises_curated_floors(tree):
+ root, amazon = tree
+ report = run(root, [amazon], WORKSPACE, NOW, _fetch, _green)
+ assert {b.unit: str(b.target) for b in report.raised} == {
+ "boto3+botocore": "1.42.0",
+ "google-cloud-bigquery": "3.9.0",
+ }
+ text = amazon.read_text()
+ assert '"boto3>=1.42.0"' in text
+ assert '"botocore>=1.42.0"' in text
+ assert '"google-cloud-bigquery>=3.9.0"' in text
+ assert "held back by <3" in report.skipped["google-cloud-storage"]
+ # not curated, so never considered
+ assert "sagemaker-studio" not in report.skipped
+
+
+def test_run_ignores_workspace_members(tree):
+ root, amazon = tree
+ (root / "pyproject.toml").write_text(
+ ROOT_CONFIG.replace('"Google_Cloud-*"', '"Google_Cloud-*",
"apache-airflow-*"')
+ )
+ report = run(root, [amazon], WORKSPACE, NOW, _fetch, _green)
+ assert all("apache-airflow" not in b.unit for b in report.raised)
+ assert '"apache-airflow-providers-google>=1.0"' in amazon.read_text()
+
+
+def test_run_skips_package_when_fetch_fails(tree):
+ root, amazon = tree
+ report = run(root, [amazon], frozenset(), NOW, lambda name:
_fetch("missing"), _green)
+ assert report.raised == []
+ assert report.skipped["google-cloud-bigquery"].startswith("PyPI metadata
unavailable")
+ assert amazon.read_text() == AMAZON
+
+
+def test_render_report():
+ bump = Bump(
+ unit="boto3+botocore",
+ packages=("boto3", "botocore"),
+ old_floors=("1.41.0",),
+ target=Version("1.42.0"),
+ edits=(Edit(path=Path("providers/amazon/pyproject.toml"), old="x",
new="y"),),
+ )
+ bad = Bump(
+ unit="google-cloud-bigquery",
+ packages=("google-cloud-bigquery",),
+ old_floors=("3.0.0",),
+ target=Version("3.9.0"),
+ edits=(),
+ )
+ text = render_report(
+ Report(raised=[bump], skipped={"pandas": "DataFrame XComs"},
rolled_back=[(bad, "boom")])
+ )
+ assert "### Dependency floors" in text
+ assert "- `boto3`, `botocore`: 1.41.0 → 1.42.0" in text
+ assert "- `pandas`: DataFrame XComs" in text
+ assert "- `google-cloud-bigquery` → 3.9.0" in text
+ assert "boom" in text
+
+
+def test_exclusion_reason_shows_path_relative_to_repository(tmp_path):
+ config = load_config(_write(tmp_path, CONFIG))
+ site = _site("boto3>=1.41,<2", str(AIRFLOW_ROOT_PATH / "providers" /
"amazon" / "pyproject.toml"))
+ reason = get_exclusion_reason("boto3", [site], config)
+ assert reason == "held back by <2 in providers/amazon/pyproject.toml"
+
+
+def test_workspace_pyprojects_skip_members_with_own_lock(tmp_path):
+ # A member with its own uv.lock (dev/breeze) is not covered by the root
resolve check,
+ # so raising its floors would leave that lock stale.
+ (tmp_path / "pyproject.toml").write_text(
+ '[tool.uv.workspace]\nmembers = [".", "providers/*", "dev/breeze"]\n'
+ )
+ for member in ("providers/amazon", "providers/google", "dev/breeze"):
+ (tmp_path / member).mkdir(parents=True)
+ (tmp_path / member / "pyproject.toml").write_text("[project]\n")
+ (tmp_path / "dev/breeze/uv.lock").write_text("")
+ assert get_workspace_pyprojects(tmp_path) == [
+ tmp_path / "pyproject.toml",
+ tmp_path / "providers/amazon/pyproject.toml",
+ tmp_path / "providers/google/pyproject.toml",
+ ]
+
+
+def test_revert_restores_only_the_edited_occurrences(tmp_path):
+ path = tmp_path / "pyproject.toml"
+ original = (
+ '[project]\ndependencies = ["google-cloud-x>=1.0"]\n'
+ '[dependency-groups]\ndev = ["google-cloud-x>=2.0"]\n'
+ )
+ path.write_text(original)
+ sites = [_site("google-cloud-x>=1.0", str(path)),
_site("google-cloud-x>=2.0", str(path))]
+ bump = build_bump("google-cloud-x", sites, Version("2.0"))
+ apply_bump(bump)
+ assert path.read_text().count('"google-cloud-x>=2.0"') == 2
+ revert_bump(bump)
+ # The requirement that was already at the target must not be lowered by
the rollback.
+ assert path.read_text() == original
+
+
+def test_apply_bump_is_all_or_nothing(tmp_path):
+ first, second = tmp_path / "a.toml", tmp_path / "b.toml"
+ first.write_text('dependencies = ["boto3>=1.41.0"]\n')
+ # The parsed requirement does not match the file text (escaped quotes), so
this edit cannot apply.
+ second.write_text('dependencies = ["boto3>=1.41.0; python_version <
\\"3.14\\""]\n')
+ bump = build_bump(
+ "boto3",
+ [_site("boto3>=1.41.0", str(first)), _site("boto3>=1.40.0;
python_version < '3.14'", str(second))],
+ Version("1.43.0"),
+ )
+ with pytest.raises(ValueError, match="not found"):
+ apply_bump(bump)
+ assert first.read_text() == 'dependencies = ["boto3>=1.41.0"]\n'
+
+
+def test_failed_apply_reverts_earlier_bumps(workspace, monkeypatch):
+ ws = workspace([])
+
+ def apply_or_fail(bump):
+ if bump.unit == "b":
+ raise ValueError("Requirement not found")
+ ws.apply(bump)
+
+ monkeypatch.setattr("upgrade_dependency_floors.apply_bump", apply_or_fail)
+ with pytest.raises(ValueError, match="not found"):
+ apply_with_rollback([_bump("a"), _bump("b")], ws.check)
+ assert ws.applied == set()
+
+
+def test_run_skips_package_with_malformed_release_data(tree):
+ root, amazon = tree
+
+ def fetch(name):
+ if name == "google-cloud-bigquery":
+ return {"3.9.0": [{"yanked": False}]}
+ return _fetch(name)
+
+ report = run(root, [amazon], WORKSPACE, NOW, fetch, _green)
+ assert report.skipped["google-cloud-bigquery"].startswith("PyPI metadata
unavailable")
+ assert {b.unit for b in report.raised} == {"boto3+botocore"}
+
+
[email protected](
+ "error",
+ [
+ pytest.param(ValueError("bad config"), id="config"),
+ pytest.param(LockAlreadyBrokenError("bad config"), id="lock"),
+ ],
+)
+def test_main_reports_why_floors_were_not_updated(tmp_path, monkeypatch,
error):
+ report_path = tmp_path / "report.md"
+ monkeypatch.setenv("DEPENDENCY_FLOORS_REPORT", str(report_path))
+
+ def fail(*args, **kwargs):
+ raise error
+
+ monkeypatch.setattr("upgrade_dependency_floors.run", fail)
+ assert main() == 1
+ text = report_path.read_text()
+ assert "### Dependency floors" in text
+ assert "Not updated: bad config" in text
+
+
+def test_render_report_lists_files_of_raised_floors():
+ amazon = AIRFLOW_ROOT_PATH / "providers" / "amazon" / "pyproject.toml"
+ google = AIRFLOW_ROOT_PATH / "providers" / "google" / "pyproject.toml"
+ bump = Bump(
+ unit="boto3",
+ packages=("boto3",),
+ old_floors=("1.41.0",),
+ target=Version("1.42.0"),
+ edits=(
+ Edit(path=google, old="a", new="b"),
+ Edit(path=amazon, old="c", new="d"),
+ Edit(path=amazon, old="e", new="f"),
+ ),
+ )
+ text = render_report(Report(raised=[bump], skipped={}, rolled_back=[]))
+ assert (
+ "- `boto3`: 1.41.0 → 1.42.0 (providers/amazon/pyproject.toml,
providers/google/pyproject.toml)"
+ in text
+ )
+
+
[email protected]("table", ["constraint-dependencies",
"override-dependencies"])
+def test_run_respects_holds_in_root_uv_settings(tree, table):
+ root, amazon = tree
+ (root / "pyproject.toml").write_text(
+ ROOT_CONFIG + f'\n[tool.uv]\n{table} = ["google-cloud-bigquery<3.5"]\n'
+ )
+ report = run(root, [amazon], WORKSPACE, NOW, _fetch, _green)
+ assert report.skipped["google-cloud-bigquery"] == f"held back by <3.5 in
[tool.uv] {table}"
+ assert '"google-cloud-bigquery>=3.0.0"' in amazon.read_text()
+
+
+MANY_GOOGLE = [f"google-cloud-p{i}" for i in range(6)]
+
+
[email protected]
+def many_tree(tmp_path):
+ (tmp_path / "pyproject.toml").write_text(ROOT_CONFIG)
+ provider = tmp_path / "pyproject.provider.toml"
+ deps = ",\n".join(f' "{name}>=1.0"' for name in MANY_GOOGLE)
+ provider.write_text(f'[project]\nname = "p"\ndependencies =
[\n{deps}\n]\n')
+ return tmp_path, provider
+
+
+def test_run_stops_asking_pypi_when_it_is_unreachable(many_tree):
+ root, provider = many_tree
+ calls = []
+
+ def unreachable(name):
+ calls.append(name)
+ raise requests.ConnectionError("connection timed out")
+
+ report = run(root, [provider], frozenset(), NOW, unreachable, _green)
+ assert len(calls) == 3
+ assert set(report.skipped) == set(MANY_GOOGLE)
+ assert report.skipped[MANY_GOOGLE[-1]] == "PyPI unreachable (3 consecutive
connection failures)"
+
+
+def test_http_errors_do_not_count_as_unreachable(many_tree):
+ root, provider = many_tree
+ calls = []
+
+ def not_found(name):
+ calls.append(name)
+ raise requests.HTTPError("404 Not Found")
+
+ run(root, [provider], frozenset(), NOW, not_found, _green)
+ assert len(calls) == len(MANY_GOOGLE)
