sjyangkevin commented on issue #54454: URL: https://github.com/apache/airflow/issues/54454#issuecomment-6753868953
Hi @Pyasma, I wonder if you're still working on this? I came across this issue and spent some time to understand context with the help of Claude. I'd like to share some findings and an approach idea, and get feedback on whether it makes sense. Happy to help on #69701, or to pick it up if you've moved on to something else. The first thing is that `PROVIDERS_COMPATIBILITY_TESTS_MATRIX` is no longer used only by the provider compat tests. Since #69671, the OpenLineage e2e compat tests read it too, and both jobs treat every entry as a published release with a PyPI wheel, a Docker image and a `constraints-<version>` tag. A branch has none of these, which is why the OpenLineage jobs fail on #69701: the branch entry turns into the image name `apache/airflow:apache/airflow:v3-3-test-python3.10`. #74997 ran into the same assumption with an unreleased `X.Y.0`. Then, I asked Claude to run the install steps in breeze against the tip of `v3-3-test`, which reports version `3.3.2`. When the provider install step doesn't pin Airflow, a provider that needs `apache-airflow>=3.4.0` makes uv quietly replace the branch build with `3.4.0` from PyPI, and the step still succeeds. The tests would then run against a release instead of the branch, without anyone noticing. With the pin, the same case fails with a clear "unsatisfiable" error, and the pinned install only takes a few seconds once the git clone is cached. Also, it seems installing Airflow from git follows the branch's own `tool.uv.sources`. This means providers like common-compat are built from the branch's outdated copy (`1.15.0`), which conflicts with `constraints-3-3` (`1.19.0`) and breaks the install. Adding `--no-sources` fixes this. Based on this, the idea is to keep the matrix for published versions only, since it's maintained as part of the release process and other jobs rely on it, and to add a separate branch-tip job instead. Selective checks would work out the branch from the highest version in the matrix (`3.4.0` gives `v3-4-test` and `constraints-3-4`), so nothing new needs updating when a new branch is cut. The job would resolve the branch to a commit SHA at the start, install Airflow from that commit with `--no-sources`, keep the Airflow pin when installing providers from `main`, and then check that the installed commit in `direct_url.json` matches the SHA. Because the branch tip moves independently of `main`, a broken backport would fail unrelated provider PRs that their authors can't fix. So it might be better to run only on canary runs, or when a PR has the `canary` or `full tests needed` label. There are a few things still not very clear. While a new minor is in beta and `vX-Y-test` exists but `X.Y.0` isn't released yet, should the job test the new branch or the previous one? Is cloning the repo on each canary run acceptable, or would it be better to build wheels from the branch once? And should this be a separate job, or share steps with the existing compat job? Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
