sjyangkevin opened a new pull request, #76957:
URL: https://github.com/apache/airflow/pull/76957
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->
<!--
Thank you for contributing!
Please provide above a brief description of the changes made in this pull
request.
Write a good git commit message following this guide:
https://chris.beams.io/posts/git-commit/
Please make sure that your code changes are covered with tests.
And in case of new features or big changes remember to adjust the
documentation.
For user-facing UI changes, please attach before/after screenshots (or a
short
screen recording) so reviewers can assess the visual impact.
Feel free to ping (in general) for the review if you do not see reaction for
a few days
(72 Hours is the minimum reaction time you can expect from volunteers) - we
sometimes miss notifications.
In case of an existing issue, reference it using one of the following:
* closes: #ISSUE
* related: #ISSUE
-->
closes: #54454
>The following illustration use 3.4.0 as an example, but it should work for
future release.
### What
Today, the provider compatibility tests only run against Airflow versions
that are already released on PyPI. Bug fixes for the next patch release are
backported to the release branch (`v3-4-test` at the moment). So if a backport
breaks a provider, we might find out after the release is out. #54454 asks for
compatibility tests against the `v3-X-test` branch tip, so we can catch this
earlier.
This PR adds a new workflow, `test-providers-branch-tip.yml`, that runs the
provider tests from `main` against Airflow built from the tip of the release
branch. It runs next to the existing compatibility jobs and does not change
them.
### How it works
1. **Find the branches.** Selective checks pass the latest release in
`PROVIDERS_COMPATIBILITY_TESTS_MATRIX` (3.4.0 today). The job lists the
`vX-Y-test` branches of the repository and keeps the ones whose `X.Y` is at
least 3.4 and that have a `constraints-X-Y` branch. Today that is only
`v3-4-test`. While the next minor release is in beta, it would be both
`v3-4-test` and `v3-5-test`. Once 3.5.0 is released and added to the matrix,
`v3-4-test` drops out. There is no new list to maintain, and the matrix keeps
only published releases, which the OpenLineage e2e compatibility tests also
rely on.
2. **Build Airflow from the branch tip.** The job checks out the branch at
the commit it found and builds the Airflow and Task SDK wheels with the
branch's own breeze, the same way a release is built, including the compiled
UI. It then checks that the wheel records that commit (`airflow/git_version`).
3. **Run the provider tests.** These are the same steps as the existing
compatibility job, except that Airflow is installed from those wheels
(`--use-airflow-version wheel`) with the branch's constraints
(`constraints-3-4`).
The issue suggests installing Airflow from the branch with git (#54070). I
tried that first, but it needs a full clone of the repository, it builds the
providers Airflow depends on from the branch's outdated copies (which conflict
with the branch constraints), and it skips the UI build that some provider
tests need. Building wheels avoids all three. Installing them from `dist` also
pins Airflow, so the resolver cannot replace the branch build with the 3.4.0
release from PyPI, which has the same version number.
### When it runs
| Event | Runs? |
|---|---|
| Canary runs on `main` (twice a day on AMD, twice a day on ARM) | Yes |
| A PR with the `full tests needed` label | Yes |
| A PR that changes `test-providers-branch-tip.yml`, like this one | Yes |
| Any other PR, including provider changes | No |
| Release branches | No |
It does not run on every provider PR because the branch tip moves on its
own. A provider change is already covered by the 3.4.0 job, and a failure on
the branch tip would usually come from a backport that the PR author cannot fix.
### Before and after, for a canary run
| | Before | After |
|---|---|---|
| Compatibility jobs against released versions | 6 (2.11.1 to 3.4.0) | 6,
unchanged |
| Branch tip jobs | None | 1 lookup (seconds), 1 build (a few minutes), 1
test job. Two builds and two test jobs while a minor release is in beta |
| Regular PR runs | No branch tip jobs | No branch tip jobs, except the
cases in the table above |
When the branch tip job fails and the 3.4.0 job in the same run passes, the
cause is in a backport since 3.4.0 (`git log 3.4.0..<sha>`).
The new job is listed in `CI passed` and `Check for timed-out CI jobs`, so a
failure shows up on the canary run and in Slack. **I did not add it to
`Finalize tests`, so that a broken backport on a release branch does not stop
`main` from publishing its images. Does this sound right to you?**
### Testing
I ran the steps of the job locally in breeze against `v3-4-test` at
`a707ed72f8`:
- Building the wheels with the branch's own breeze took about two and a half
minutes. The wheels include the compiled UI, and `git_version` matches the
commit.
- The `standard` provider tests had 1227 passed and 9 failed. The same 9
tests failed against released 3.4.0, because my local run did not include the
`openlineage` provider. So the branch tip did not break anything there.
- The `fab` provider tests, which start the API app and need the compiled
UI, had 747 passed.
- As a negative control, I renamed a keyword argument of
`upsert_hitl_detail()` in a local commit on top of the branch tip. Only the
HITL tests failed, with a `TypeError`. This shows the job catches a breaking
backport, and that the tests really used the branch wheels.
I also ran the branch lookup against the GitHub API with different latest
releases, to check that it compares versions as numbers (`v2-10-test` comes
after `v2-9-test`) and skips branches without a constraints branch. The new
selective-check outputs have unit tests, and `test_selective_checks.py` and
`test_verification_plan.py` pass. The workflow itself can only run on GitHub,
and this PR changes it, so its CI runs the new job.
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR,
please
change below checkbox to `[X]` followed by the name of the tool, uncomment
the "Generated-by".
-->
- [X] Yes (please specify the tool below)
Generated-by: [Claude Code] following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
---
* Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
for more information. Note: commit author/co-author name and email in commits
become permanently public when merged.
* For fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment:
`{pr_number}.significant.rst`, in
[airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
You can add this file in a follow-up commit after the PR is created so you
know the PR number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
