namanjain24-sudo opened a new pull request, #73129:
URL: https://github.com/apache/airflow/pull/73129
The lowest-deps provider tests sync with `--resolution lowest-direct`, but
every other provider is a workspace member, so cross-provider dependencies
always came from the checkout. A provider declaring `common-compat>=1.8.0` was
really tested against the current `common-compat`.
uv can't be told to skip that for a few packages here:
`--no-sources-package` fails in this workspace ("references a workspace in
tool.uv.sources but is not a workspace member"), and dropping providers from
the workspace breaks the lock while some providers pin versions that only exist
as release candidates. So the lowering now happens right after the sync:
- `scripts/in_container/lower_cross_provider_dependencies.py` reads the
tested provider's `[project].dependencies`. For each provider dependency with a
lower bound it picks the lowest final (non-yanked) release that every installed
distribution accepts, looking up releases on pypi.org, then repeats with the
pinned releases' own requirements until the pins stop changing. Lines with `#
use next version`, dependencies without a lower bound, and ones where only an
rc satisfies the bounds keep the workspace version. It logs why each dependency
was pinned or kept.
- `entrypoint_ci.sh` installs those pins and runs `uv pip check`. They are
installed with their dependencies on purpose: an old `cncf-kubernetes` needs an
old `kubernetes` client, and `--no-deps` would leave a broken environment.
Running this over all providers found two lower bounds that don't work,
fixed here so the job stays green:
- `apache.flink` required `cncf-kubernetes>=5.1.0`, but 36 of the 39
releases from 5.1.0 to 8.4.2, including every one from 7.4.0 on, import
`airflow.compat`, which Airflow 3 no longer has. flink's tests fail to import
on those I tried (5.1.0, 7.4.0, 8.0.0–8.1.1, 8.3.0–8.4.2). 9.0.0 is the first
release without that import, and the tests pass on 9.0.0, 10.0.0, 10.5.0,
10.6.0 and 10.12.2. Bumped to `>=9.0.0`.
- `airflow-core` requires `common-sql>=1.28.1`, but common-sql releases
before 1.30.2 import `airflow.utils.module_loading`, removed in 3.2.0 by
#59139. Bumped to `>=1.30.2`, the same way the pre-installed minimums were
bumped in #56208. This also covers `apache.arrow` and `slack`, whose own bounds
are lower.
What I ran:
| | |
|---|---|
| Every provider, locally (lowest-direct sync, lowering, `uv pip check`,
tests; failing ones compared with a run without lowering) | 107 providers, 99
of them got pins (127 in total), `uv pip check` clean for every provider that
synced. `apache.arrow` was the only failure caused by the lowering, fixed by
the core bump above. Every other test that failed with the lowering also failed
in a run without it. Providers that could not sync or run their tests on my Mac
(native libraries, an SSH tunnel test) are in the Breeze row. |
| Breeze `--force-lowest-dependencies` | amazon, google, ibm.db2, odbc,
sftp, ssh, vespa on this branch; apache.drill, apache.flink, ftp, postgres on
an earlier revision of it (before the common-sql bump, which doesn't change
their pins). All pass, `uv pip check` clean |
| `scripts/tests/in_container/test_lower_cross_provider_dependencies.py` |
22 passed |
closes: #60888
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (please specify the tool below)
Generated-by: a Gen-AI coding assistant, following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions).
I reviewed the change and ran the checks above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
