kacpermuda opened a new pull request, #69212:
URL: https://github.com/apache/airflow/pull/69212
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->
Add framework for providers E2E tests and implement OpenLineage provider e2e
tests to Airflow CI
The OpenLineage provider emits lineage events when Dags run, but until now
there was no automated way in Airflow CI to verify those events are actually
correct against a real, deployed Airflow stack. This PR introduces that
capability: a self-contained e2e test harness that spins up Airflow via
docker-compose, runs the provider system-test Dags, and asserts the OpenLineage
events emitted by the transport match expected payloads. The tests run on
demand (via `workflow_dispatch`) and are triggered automatically by selective
checks when the `openlineage` or `common` providers, the `providers-e2e-tests`
harness, or related files change.
The approach follows the same pattern already established by the Task SDK
integration tests and the Airflow e2e tests: a standalone directory under
`providers-e2e-tests/openlineage/` with its own `pyproject.toml`,
`docker-compose.yaml`, and pytest suite, invoked through a new `breeze testing
providers-e2e-tests <provider>` command.
More providers can be added in the future if needed. These OL test do not
rely on any external service, so they can easily be run in CI (no db, external
api, google/amazon/azure service needed).
**What's included:**
- `providers-e2e-tests/` — new top-level directory; each provider gets its
own subdirectory with a
docker-compose stack, tests, and `pyproject.toml`.
- `breeze testing providers-e2e-tests <provider>` — spins up the stack and
runs pytest.
- Two modes: default PROD image, or `--airflow-version <ver>` to test older
Airflow + current providers
from main (builds `apache/airflow:<ver>` + provider wheels). Required
providers are declared in each
provider's `pyproject.toml` under `[tool.e2e-tests] required-providers` —
nothing hardcoded in breeze.
- OpenLineage implementation: postgres + API server + scheduler +
dag-processor + triggerer + worker.
DAGs come from the existing
`providers/openlineage/tests/system/openlineage/` — `prepare_dags.py`
copies and strips their pytest-only footers at runtime. Events are
captured via `VariableTransport`
(stores OL events in Airflow Variables); each DAG's
`OpenLineageTestOperator` validates them — no
external backend needed. A run that ends `success` means lineage matched.
- Selective checks trigger the suite on openlineage / common-providers
changes. Core and task-sdk changes are not wired in - a regression there
wouldn't trigger these tests on the PR, only on the next canary run. Adding
core/task-sdk to the file group would work technically (PROD image is already
built when those change via other test paths), but it would widen the trigger
surface significantly. Leaving it for discussion.
- CI: PROD and compat matrix suite on every relevant PR and canary/main; the
compat matrix (3.0.6 / 3.1.8 / 3.2.2) is reused directly from
PROVIDERS_COMPATIBILITY_TESTS_MATRIX in breeze's global_constants.py - the same
source as the provider unit compat tests - so adding a new Airflow release
there automatically picks up a new e2e compat run. The 2.x entry is skipped via
a startsWith('3.') condition since the e2e stack requires the Airflow 3 API
server (we will bump min AF version for providers soon anyway, so did not waste
time making it work for AF2).
I tried to add a new breeze command for this, but it's first time ever for
me contributing in this area, so it may not be ideal, tried to follow an
example of current code.
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (please specify the tool below)
Generated-by: Claude Sonnet 4.6 following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
---
* Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
for more information. Note: commit author/co-author name and email in commits
become permanently public when merged.
* For fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment:
`{pr_number}.significant.rst`, in
[airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
You can add this file in a follow-up commit after the PR is created so you
know the PR number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
