potiuk commented on code in PR #56610:
URL: https://github.com/apache/airflow/pull/56610#discussion_r2429383002
##########
dev/breeze/doc/ci/05_workflows.md:
##########
@@ -118,98 +120,273 @@ trigger the `canary` run manually via
`workflow-dispatch` mechanism.
# Workflows
-A general note about cancelling duplicated workflows: for `Tests` and `CodeQL`
workflows,
-we use the `concurrency` feature of GitHub actions to automatically cancel
"old" workflow runs of
-each type. This means that if you push a new commit to a branch or to a pull
-request while a workflow is already running, GitHub Actions will automatically
cancel the
-old workflow run.
+Apache Airflow's CI system has evolved into what we call a composite workflow
architecture. This approach breaks down our testing process into manageable,
reusable pieces that work together to validate code changes.
-## Differences for `main` and `v*-*-test` branches
+## Workflow Architecture Overview
-The type of tests executed varies depending on the version or branch
-being tested. For the "main" development branch, we run all tests to
-maintain the quality of Airflow. However, when releasing patch-level
-updates on older branches, we only run a subset of tests. This is
-because older branches are used exclusively for releasing Airflow and
-its corresponding image, not for releasing providers or Helm charts,
-so all those tests are skipped there by default.
+Our CI system runs on two main workflows that handle different processor
architectures:
-This behaviour is controlled by `default-branch` output of the
-build-info job. Whenever we create a branch for an older version, we update
-the `AIRFLOW_BRANCH` in `airflow_breeze/branch_defaults.py` to point to
-the new branch. In several places, the selection of tests is
-based on whether this output is `main`. They are marked in the "Release
branches" column of
-the table below.
+- **`ci-amd.yml`** - Handles testing on AMD64 systems (the most common
architecture)
+- **`ci-arm.yml`** - Runs the same tests on ARM64 systems for compatibility
-## Tests Workflow
+These workflows don't contain all the testing logic themselves. Instead, they
call smaller, specialized workflows called composite workflows. Each composite
workflow focuses on one specific area of testing.
-This workflow is a regular workflow that performs all checks of Airflow code.
The `main` and `v*-*-test`
-pushes are `canary` runs.
+### Understanding Composite Workflows
-| Job | Description
| PR | main | v*-*-test |
+A composite workflow is essentially a reusable piece of our CI pipeline.
Rather than having one massive workflow file with hundreds of lines, we split
the work into focused components:
+
+- Basic validation happens in one workflow
+- Image building gets its own workflow
+- Database testing has dedicated workflows
+- Kubernetes testing runs separately
+
+Each composite workflow is stored in `.github/workflows/` and gets called
using GitHub's `uses: ./.github/workflows/...` syntax. When we need to modify
how database tests work, we only touch that specific workflow file.
+
+### Workflow Cancellation
+
+GitHub Actions includes a concurrency feature that we use to prevent wasted
resources. If you push new commits while tests are already running, the system
cancels the old test run and starts over with your latest code. This saves time
and compute resources.
+
+## Branch-Specific Behavior
+
+Different branches in our repository serve different purposes, so they get
different levels of testing.
+
+### Main Branch Testing
+
+The `main` branch undergoes the most extensive testing since it serves as the
primary development branch. All available test suites execute on this branch:
+
+- Core Airflow functionality tests
+- Provider package tests
+- Integration tests with external systems
+- Helm chart validation
+- Kubernetes deployment tests
+- Documentation building and validation
+
+We run everything on `main` because we want to catch problems as early as
possible in the development cycle.
+
+### Release Branch Testing
+
+Release branches (named `v*-*-test`) get a more focused set of tests. These
branches are used only for releasing Airflow core and Docker images, not for
provider packages or Helm charts. So we skip tests that aren't relevant:
+
+- Core Airflow tests still run
+- Image building and validation continues
+- Provider-specific tests are skipped
+- Helm and Kubernetes tests are skipped
+
+This approach saves time and resources while still ensuring release quality.
+
+### Branch Detection Logic
+
+The system determines which tests to run through the `build-info` job. This
job checks the `AIRFLOW_BRANCH` setting in `airflow_breeze/branch_defaults.py`
to understand what branch type it's working with, then enables or disables test
groups accordingly.
+
+## Tests Workflow Structure
+
+The Tests workflow handles all code validation for Apache Airflow. When
changes get pushed to `main` or release branches, these runs become "canary"
builds that also update shared infrastructure like dependency constraints and
Docker images.
+
+### Workflow Execution Flow
+
+Tests run in a specific sequence, with each stage building on the results of
previous stages:
+
+#### 1. Build Info
+
+The build-info job examines what files changed in your pull request and
decides which tests need to run. If you only changed documentation, it might
skip Python tests entirely. If you modified provider code, it focuses on
provider-related testing. This selective approach saves time and resources.
+
+#### 2. Basic Tests
+
+Before running expensive tests, we validate basic functionality:
+
+- Breeze development environment tests
+- React UI component tests
+- Code formatting and style checks
+- Dependency upgrade monitoring
+
+These quick checks catch obvious problems early.
+
+#### 3. Image Building
+
+We build the Docker environments needed for testing:
+
+- CI images optimized for running tests
+- Production images that match what users download
+- Dependency constraint files that lock versions for consistent testing
+
+Having standardized environments ensures tests behave predictably.
+
+#### 4. Core Testing
+
+The main testing phase runs multiple test suites in parallel:
+
+- Database tests against PostgreSQL, MySQL, and SQLite
+- Non-database tests for core logic
+- Provider package tests for external integrations
+- Integration tests that verify component interactions
+
+Parallel execution speeds up the overall process.
+
+#### 5. Specialized Testing
+
+Additional tests verify specific deployment scenarios:
+
+- Kubernetes integration tests
+- Helm chart deployment tests
+- Task SDK and CLI tool validation
+
+These ensure Airflow works correctly in various real-world environments.
+
+#### 6. Finalization
+
+Successful test runs trigger several cleanup and publishing tasks:
+
+- Updated Docker images get pushed to the registry
+- Constraint files get updated with tested dependency versions
+- Test warnings and issues get summarized
+- Artifacts get prepared for future builds
+
+### Composite Workflow Groups
+
+Here's what each workflow group does and when it runs:
+
+| Workflow Group | Purpose
| PR | main | v*-*-test |
+|------------------------------------|------------------------------------------------------------|---------|---------|-----------|
+| **Build Info** | Analyzes changes and determines which
tests to run | Yes | Yes | Yes |
+| **Basic Tests** | Runs quick validation: Breeze, UI,
static checks | Yes | Yes | Yes |
+| **CI Image Build** | Builds Docker images for testing
| Yes | Yes | Yes |
+| **Additional CI Image Checks** | Validates CI images meet quality
standards | Yes | Yes | Yes |
+| **Generate Constraints** | Creates dependency version lock files
| Yes | Yes | Yes |
+| **CI Image Checks** | Runs static analysis and builds docs
(AMD only) | Yes | Yes | Yes |
Review Comment:
This table (and next) seems to broken - alignment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
