This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new 82b3c11277 Improve CI.rst document clarity. (#36181)
82b3c11277 is described below
commit 82b3c1127790a836f364cc45d13b948fe0f30f45
Author: UncertainBadg3r <[email protected]>
AuthorDate: Wed Dec 13 02:43:56 2023 -0500
Improve CI.rst document clarity. (#36181)
* Improve document clarity.
* Remove whitespace at end of lines.
* Use double-backticks for code.
---
CI.rst | 125 ++++++++++++++++++++++++++++++++++++-----------------------------
1 file changed, 69 insertions(+), 56 deletions(-)
diff --git a/CI.rst b/CI.rst
index 41c7d0b17f..fdedb0cd99 100644
--- a/CI.rst
+++ b/CI.rst
@@ -38,20 +38,27 @@ of how Airflow CI works.
GitHub Actions runs
-------------------
-Our builds on CI are highly optimized. They utilise some of the latest
features provided by GitHub Actions
-environment that make it possible to reuse parts of the build process across
different Jobs.
-
-Big part of our CI runs use Container Images. Airflow has a lot of
dependencies and in order to make
-sure that we are running tests in a well configured and repeatable
environment, most of the tests,
-documentation building, and some more sophisticated static checks are run
inside a docker container
-environment. This environment consist of two types of images: CI images and
PROD images. CI Images
-are used for most of the tests and checks where PROD images are used in the
Kubernetes tests.
-
-In order to run the tests, we need to make sure that the images are built
using latest sources and that it
-is done quickly (full rebuild of such image from scratch might take ~15
minutes). Therefore optimisation
-techniques have been implemented that use efficiently cache from the GitHub
Docker registry - in most cases
-this brings down the time needed to rebuild the image to ~4 minutes. In some
cases (when dependencies change)
-it can be ~6-7 minutes and in case base image of Python releases new
patch-level, it can be ~12 minutes.
+Our CI builds are highly optimized, leveraging the latest features provided
+by the GitHub Actions environment to reuse parts of the build process across
+different jobs.
+
+A significant portion of our CI runs utilize container images. Given that
+Airflow has numerous dependencies, we use Docker containers to ensure tests
+run in a well-configured and consistent environment. This approach is used
+for most tests, documentation building, and some advanced static checks.
+The environment comprises two types of images: CI images and PROD images.
+CI images are used for most tests and checks, while PROD images are used for
+Kubernetes tests.
+
+To run the tests, we need to ensure that the images are built using the
+latest sources and that the build process is efficient. A full rebuild of
+such an image from scratch might take approximately 15 minutes. Therefore,
+we've implemented optimization techniques that efficiently use the cache
+from the GitHub Docker registry. In most cases, this reduces the time
+needed to rebuild the image to about 4 minutes. However, when
+dependencies change, it can take around 6-7 minutes, and if the base
+image of Python releases a new patch-level, it can take approximately
+12 minutes.
Container Registry used as cache
--------------------------------
@@ -105,7 +112,7 @@ The image names follow the patterns (except the Python
image, all the images are
https://ghcr.io/ in ``apache`` organization.
The packages are available under (CONTAINER_NAME is url-encoded name of the
image). Note that "/" are
-supported now in the ``ghcr.io`` as apart of the image name within ``apache``
organization, but they
+supported now in the ``ghcr.io`` as a part of the image name within the
``apache`` organization, but they
have to be percent-encoded when you access them via UI (/ = %2F)
``https://github.com/apache/airflow/pkgs/container/<CONTAINER_NAME>``
@@ -192,29 +199,33 @@ When you are running the CI jobs in GitHub Actions,
GITHUB_TOKEN is set automati
CI run types
============
-The following CI Job run types are currently run for Apache Airflow (run by
ci.yaml workflow)
-and each of the run types has different purpose and context.
+The Apache Airflow project utilizes several types of Continuous Integration
(CI)
+jobs, each with a distinct purpose and context. These jobs are executed by the
+``ci.yaml`` workflow.
-Besides the regular "PR" runs we also have "Canary" runs that are able to
detect most of the
-problems that might impact regular PRs early, without necessarily failing all
PRs when those
-problems happen. This allows to provide much more stable environment for
contributors, who
-contribute their PR, while giving a chance to maintainers to react early on
problems that
-need reaction, when the "canary" builds fail.
+In addition to the standard "PR" runs, we also execute "Canary" runs.
+These runs are designed to detect potential issues that could affect
+regular PRs early on, without causing all PRs to fail when such problems
+arise. This strategy ensures a more stable environment for contributors
+submitting their PRs. At the same time, it allows maintainers to proactively
+address issues highlighted by the "Canary" builds.
Pull request run
----------------
-Those runs are results of PR from the forks made by contributors. Most builds
for Apache Airflow fall
-into this category. They are executed in the context of the "Fork", not main
-Airflow Code Repository which means that they have only "read" permission to
all the GitHub resources
-(container registry, code repository). This is necessary as the code in those
PRs (including CI job
-definition) might be modified by people who are not committers for the Apache
Airflow Code Repository.
+These runs are triggered by pull requests from contributors' forks. The
majority of
+Apache Airflow builds fall into this category. They are executed in the
context of
+the contributor's "Fork", not the main Airflow Code Repository, meaning they
only have
+"read" access to all GitHub resources, such as the container registry and code
repository.
+This is necessary because the code in these PRs, including the CI job
definition,
+might be modified by individuals who are not committers to the Apache Airflow
Code Repository.
-The main purpose of those jobs is to check if PR builds cleanly, if the test
run properly and if
-the PR is ready to review and merge. The runs are using cached images from the
Private GitHub registry -
-CI, Production Images as well as base Python images that are also cached in
the Private GitHub registry.
-Also for those builds we only execute Python tests if important files changed
(so for example if it is
-"no-code" change, no tests will be executed.
+The primary purpose of these jobs is to verify if the PR builds cleanly, if
the tests
+run correctly, and if the PR is ready for review and merge. These runs utilize
cached
+images from the Private GitHub registry, including CI, Production Images, and
base
+Python images. Furthermore, for these builds, we only execute Python tests if
+significant files have changed. For instance, if the PR involves a "no-code"
change,
+no tests will be executed.
Regular PR builds run in a "stable" environment:
@@ -232,30 +243,32 @@ and has WRITE access to the GitHub Container Registry.
Canary run
----------
-This is the flow that happens when a pull request is merged to the "main"
branch or pushed to any of
-the "v2-*-test" branches. The "Canary" run attempts to upgrade dependencies to
the latest versions
-and quickly pushes a preview of cache the CI/PROD images to the GitHub
Registry - so that pull requests
-can quickly use the new cache - this is useful when Dockerfile or installation
scripts change because such
-cache will already have the latest Dockerfile and scripts pushed even if some
tests will fail.
-When successful, the run updates the constraints files in the
"constraints-main" branch with the latest
-constraints and pushes both cache and latest CI/PROD images to the GitHub
Registry.
-
-When "Canary" build fails, it's often a sign that some of our dependencies
released a new version that
-is not compatible with current tests or Airflow code, Also it might mean that
a breaking change has been
-merged to "main". Both cases should be addressed quickly by the maintainers.
The "broken main" by our code
-should be fixed quickly, while the "broken dependencies" can take a bit of
time to fix as until the tests
-succeeds, constraints will not be updated, which means that regular PRs will
continue using the old version
-of dependencies that already passed one of the previous "Canary" runs.
-
+This workflow is triggered when a pull request is merged into the "main"
branch or pushed to any of
+the "v2-*-test" branches. The "Canary" run aims to upgrade dependencies to
their latest versions
+and promptly pushes a preview of the CI/PROD image cache to the GitHub
Registry. This allows pull
+requests to quickly utilize the new cache, which is particularly beneficial
when the Dockerfile or
+installation scripts have been modified. Even if some tests fail, this cache
will already include the
+latest Dockerfile and scripts.Upon successful execution, the run updates the
constraint files in the
+"constraints-main" branch with the latest constraints and pushes both the
cache and the latest CI/PROD
+images to the GitHub Registry.
+
+If the "Canary" build fails, it often indicates that a new version of our
dependencies is incompatible
+with the current tests or Airflow code. Alternatively, it could mean that a
breaking change has been
+merged into "main". Both scenarios require prompt attention from the
maintainers. While a "broken main"
+due to our code should be fixed quickly, "broken dependencies" may take longer
to resolve. Until the tests
+pass, the constraints will not be updated, meaning that regular PRs will
continue using the older version
+of dependencies that passed one of the previous "Canary" runs.
Scheduled runs
--------------
-This is the flow that happens when a scheduled run is triggered. The
"scheduled" workflow is aimed to
-run regularly (overnight). Scheduled run is generally the same as "Canary"
run, with the difference
-that the image is build always from the scratch and not from the cache. This
way we can check that no
-"system" dependencies in debian base image have changed and that the build is
still reproducible.
-No separate diagram is needed for scheduled run as it is identical to that of
"Canary" run.
+The "scheduled" workflow, which is designed to run regularly (typically
overnight),
+is triggered when a scheduled run occurs. This workflow is largely identical
to the
+"Canary" run, with one key difference: the image is always built from scratch,
not
+from a cache. This approach ensures that we can verify whether any "system"
dependencies
+in the Debian base image have changed, and confirm that the build process
remains reproducible.
+Since the process for a scheduled run mirrors that of a "Canary" run, no
separate diagram is
+necessary to illustrate it.
Workflows
=========
@@ -310,10 +323,10 @@ that triggered it.
Differences for main and release branches
-----------------------------------------
-There are a few differences of what kind of tests are run, depending on which
version/branch the tests are executed for.
-While all our tests run for the "main" development branch to keep Airflow in
check, only a subset of those tests is run
-in older branches when we are releasing patch-level releases. This is because
we never use old branches to release
-providers and helm charts, we only use them to release Airflow and Airflow
image.
+The type of tests executed varies depending on the version or branch under
test. For the "main" development branch,
+we run all tests to maintain the quality of Airflow. However, when releasing
patch-level updates on older
+branches, we only run a subset of these tests. This is because older branches
are exclusively used for releasing
+Airflow and its corresponding image, not for releasing providers or helm
charts.
This behaviour is controlled by ``default-branch`` output of the build-info
job. Whenever we create a branch for old version
we update the ``AIRFLOW_BRANCH`` in ``airflow_breeze/branch_defaults.py`` to
point to the new branch and there are a few
