shahar1 opened a new pull request, #76722:
URL: https://github.com/apache/airflow/pull/76722
### Why
A change to the Task SDK selects the tests of all providers. Thus each such
PR also runs the provider compatibility tests: the `Compat` jobs of `Provider
distributions tests`. These jobs run the provider tests against six released
Airflow versions (2.11.1 to 3.4.0). Each job takes about 35 minutes.
These jobs cannot see the change:
- The job installs `apache-airflow==<version>` from PyPI. For Airflow 3, the
released `apache-airflow-task-sdk` comes from PyPI as its dependency. The job
builds only the provider wheels from the PR.
- Breeze forces `--mount-sources remove` for a released Airflow version
(`params/shell_params.py`). `scripts/ci/docker-compose/remove-sources.yml` then
hides `airflow-core/src` and `task-sdk/src` from the container.
- The provider tests do not collect the Task SDK tests, and they do not read
the core or Task SDK docs.
Thus the result for such a PR is the same as the result on `main`.
The data comes from the 2,163 commits merged to `main` from 2026-09-08 to
2026-11-08, and from the jobs of 3,754 failed `Tests (AMD)` PR runs (3,137
attempts) in that window. 14,451 PR runs ran in the window (237 a day).
### How
A new selective-checks output, `run-providers-compatibility-tests`, is
`false` when each changed file is one of these:
- A Python file in `airflow-core/src/` or `task-sdk/src/`, except the
version files `airflow/__init__.py` and `airflow/sdk/__init__.py`. Breeze reads
those two files on the host.
- A Python file in `task-sdk/tests/`.
- A document or an image (`rst`, `md`, `txt`, `png`, `gif`, `svg`, `jpg`,
`jpeg`, `puml`, `md5sum`, `csv`) in `airflow-core/docs/` or `task-sdk/docs/`.
- An `airflow-core/newsfragments/*.rst` file.
Any other file keeps the compatibility jobs. Examples:
- A non-Python file in the source trees. Breeze reads
`provider_info.schema.json` on the host.
- `airflow-core/tests/`. The test plugin reads
`airflow-core/tests/unit/{plugins,dags}`.
- `shared/`. The job installs the local shared distributions.
- `devel-common/` and `.github/`.
- Other files in the docs trees, such as a nested `.pre-commit-config.yaml`.
The job installs the prek hooks on the host before Breeze hides the sources.
The output is `true` on full-test runs, which include the scheduled canary
runs, and when `upgrade_to_newer_dependencies` is `true`.
`test-providers.yml` runs the `providers-compatibility-tests-matrix` job
only when the output is `true`. `ci-amd.yml` and `ci-arm.yml` pass it. The
wheel and sdist tests and the other provider test jobs do not change. `breeze
verify` lists the output as not runnable locally.
### Expected result
Over the 2,163 commits merged to `main`, the gate skips the compatibility
jobs for 50 commits. 46 of them changed the Task SDK. That is 300 jobs, about
18 runner-hours and about 33 jobs a day.
A second check uses the PR runs directly. 394 failed PR run attempts ran the
compatibility jobs for all providers. 23 of them were full-test runs. The gate
skips 45 attempts and 8,134 job-minutes. That is 24% of the job-minutes of
those 45 attempts.
Recall: 442 failed PR run attempts had a failed compatibility job. The gate
skips 10 of them. Each failing test also failed in 2 to 11 runs on other
branches in the same weeks:
- the OpenLineage stalled-task test
- `TestSqlBranch` in `common.sql`, with `Session already set for this task`
from Airflow 2.11.1
- the vendor contract test of `common.ai`
- the Docker decorator test
The gate skips no failure that only the PR caused.
### Risk
The argument depends on how the job runs. If the compatibility job starts to
install the Task SDK or `airflow-core` from the PR sources, or stops hiding the
source trees, the gate becomes wrong. A comment at the new file group in
`selective_checks.py` records this.
A failure that the gate skips on a PR still shows in the next canary run,
which keeps every job.
### Verification
| Check | Result |
|---|---|
| `test_selective_checks.py` and `test_verification_plan.py` | 474 passed,
including 13 new cases |
| Breeze test suite (`breeze verify`) | 2,615 passed. 7 failed on this
machine, and the same 7 fail on the parent commit: the S3 mocks, the shim
version check and the reproducible build |
| Adversarial review, Copilot CLI with `gpt-6.1-sol` | Round 1 found a gap
(a nested prek config in the docs trees); the docs rule now accepts only
document types. GO on the final commit |
| `prek` pre-commit and manual stages | Passed |
| Replay of the 2,163 commits and of the cached failed PR runs | See
"Expected result" |
The workflow changes have not run in CI. This PR changes
`.github/workflows/*.yml`, so its own run uses the full matrix and must confirm
them.
Stacked on #76690, #76686 and #76685. The replay scripts and their output
are not part of the PR.
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes — Claude Code (Opus 5.5)
Generated-by: Claude Code (Opus 5.5) following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
