This is an automated email from the ASF dual-hosted git repository.
jscheffl pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new d9b560cd2df Helm Chart Documentation Update (#66970)
d9b560cd2df is described below
commit d9b560cd2df484285b0377cbed73f2ac1267c664
Author: Przemysław Mirowski <[email protected]>
AuthorDate: Fri May 15 23:35:35 2026 +0200
Helm Chart Documentation Update (#66970)
* Finished workers sep & misc
* Parametrize helm/k8s version in doc
* Add deprecated parameters section
* Update providers versions to match Airflow support
* Parametrize settings resources doc
* Misc in upgrading airflow 3 doc
* Match min helm chart ver with CI
* Add newsfragment
* Template helm & depend k8s and helm on CI versions
---
chart/docs/conf.py | 9 ++++
chart/docs/customizing-labels.rst | 3 +-
chart/docs/customizing-workers.rst | 28 ++++++++-----
chart/docs/extending-the-chart.rst | 36 +++++++++-------
chart/docs/index.rst | 23 +++++++----
chart/docs/parameters-ref.rst | 40 +++++++++++++++++-
chart/docs/production-guide.rst | 11 +++--
chart/docs/quick-start.rst | 9 ++--
chart/docs/setting-resources-for-containers.rst | 55 ++++++++-----------------
chart/docs/upgrading-to-airflow-3.rst | 30 ++++++--------
chart/newsfragments/66970.significant.rst | 1 +
11 files changed, 146 insertions(+), 99 deletions(-)
diff --git a/chart/docs/conf.py b/chart/docs/conf.py
index 97e2c9dcfe7..03b9c014f86 100644
--- a/chart/docs/conf.py
+++ b/chart/docs/conf.py
@@ -38,6 +38,10 @@ import re
from typing import Any
import yaml
+from airflow_breeze.global_constants import (
+ ALLOWED_KUBERNETES_VERSIONS,
+ HELM_VERSION,
+)
from docs.utils.conf_constants import (
AIRFLOW_FAVICON_PATH,
AIRFLOW_REPO_ROOT_PATH,
@@ -300,6 +304,11 @@ jinja_contexts = {
"package_name": PACKAGE_NAME,
"package_version": PACKAGE_VERSION,
},
+ "global_ctx": {
+ "package_version": PACKAGE_VERSION,
+ "min_k8s_version": ALLOWED_KUBERNETES_VERSIONS[0],
+ "helm_version": HELM_VERSION,
+ },
}
diff --git a/chart/docs/customizing-labels.rst
b/chart/docs/customizing-labels.rst
index c31e6cb0893..70fe36fd1c2 100644
--- a/chart/docs/customizing-labels.rst
+++ b/chart/docs/customizing-labels.rst
@@ -21,7 +21,8 @@ Customizing Labels and Annotations for Pods
Customizing Pod Labels
----------------------
-The Helm Chart allows you to customize labels for your Airflow objects. You
can set global labels that apply to all objects and pods defined in the chart,
as well as component-specific labels for individual Airflow components.
+The Helm Chart allows you to customize labels for your Airflow objects. You
can set global labels that apply to all objects and pods defined in the chart,
+as well as component-specific labels for individual Airflow components.
Global Labels
~~~~~~~~~~~~~
diff --git a/chart/docs/customizing-workers.rst
b/chart/docs/customizing-workers.rst
index df4680d7741..9a201eb5111 100644
--- a/chart/docs/customizing-workers.rst
+++ b/chart/docs/customizing-workers.rst
@@ -19,20 +19,18 @@ Customizing Workers
===================
Both ``CeleryExecutor`` and ``KubernetesExecutor`` workers can be highly
customized with the :ref:`workers parameters <parameters:workers>`.
-For example, to set resources on workers:
+For example, to set resources on workers for CeleryExecutor:
.. code-block:: yaml
:caption: values.yaml
workers:
- resources:
- requests:
- cpu: 1
- limits:
- cpu: 1
-
-One notable exception for ``KubernetesExecutor`` is that the default
anti-affinity applied to ``CeleryExecutor`` workers to spread them across nodes
-is not applied to ``KubernetesExecutor`` workers, as there is no reason to
spread out per-task workers.
+ celery:
+ resources:
+ requests:
+ cpu: 1
+ limits:
+ cpu: 1
Custom ``pod_template_file``
----------------------------
@@ -43,8 +41,7 @@ worker types for ``KubernetesExecutor`` or if you need to
customize something no
.. note::
- Some configuration options between Celery and Kubernetes workers can be
overwritten by new ``workers.celery`` and ``workers.kubernetes`` sections.
- Implementation of ``workers.celery`` and ``workers.kubernetes`` is not yet
fully completed.
+ Configuration options between Celery and Kubernetes workers can be
overwritten by ``workers.celery`` and ``workers.kubernetes`` sections.
As an example, let's say you want to set ``priorityClassName`` on your workers:
@@ -71,3 +68,12 @@ As an example, let's say you want to set
``priorityClassName`` on your workers:
priorityClassName: high-priority
containers:
- name: base
+
+The same can be achieved with default `pod_template_file`_ by overriding the
``priorityClassName`` option for KubernetesExecutor like:
+
+.. code-block:: yaml
+ :caption: values.yaml
+
+ workers:
+ kubernetes:
+ priorityClassName: high-priority
diff --git a/chart/docs/extending-the-chart.rst
b/chart/docs/extending-the-chart.rst
index 930743bc62b..e368e019c20 100644
--- a/chart/docs/extending-the-chart.rst
+++ b/chart/docs/extending-the-chart.rst
@@ -21,7 +21,7 @@ Extending the Chart
The Airflow Helm Chart can be easily extended by creating a custom chart which
will depend on the Airflow chart.
That can be useful in cases where there is a need for custom templates
deployment (e.g. maintenance CronJobs),
which are not directly related to the Airflow Helm Chart and should not be
added to it in the source repository.
-During installation of custom chart, the Airflow chart will also be installed
too.
+During installation of custom chart, the Airflow chart will also be installed.
You can extend the official Airflow chart by applying the following steps.
@@ -55,17 +55,19 @@ This will give you the ability to add your custom templates
without the need to
In order to add the Airflow chart as a dependency (often called ``subcharts``)
to your chart,
add the following lines to your ``Chart.yaml`` file:
-.. code-block:: yaml
- :caption: Chart.yaml
+.. jinja:: global_ctx
+
+ .. code-block:: yaml
+ :caption: Chart.yaml
- dependencies:
- - name: airflow
- version: 1.11.0
- repository: https://airflow.apache.org
+ dependencies:
+ - name: airflow
+ version: {{ package_version }}
+ repository: https://airflow.apache.org
.. note::
- Make sure you have already added the Airflow repo locally by running:
``helm repo add apache-airflow https://airflow.apache.org``.
+ Make sure that you have already added the Airflow repo locally by running:
``helm repo add apache-airflow https://airflow.apache.org``.
.. tip::
@@ -76,14 +78,16 @@ Adding the Airflow chart as a dependency means that it will
be deployed together
You can disable the installation of Airflow by adding the ``condition`` field
to the ``dependencies`` section
like in the example below:
-.. code-block:: yaml
- :caption: Chart.yaml
+.. jinja:: global_ctx
+
+ .. code-block:: yaml
+ :caption: Chart.yaml
- dependencies:
- - name: airflow
- version: 1.11.0
- repository: https://airflow.apache.org
- condition: airflow.enabled
+ dependencies:
+ - name: airflow
+ version: {{ package_version }}
+ repository: https://airflow.apache.org
+ condition: airflow.enabled
This will check if the value of ``airflow.enabled`` inside your
``values.yaml`` is ``true``.
If it is, the Airflow chart will be deployed together with your custom chart.
@@ -101,7 +105,7 @@ you can download it by running the following command:
.. note::
- Make sure you are inside the directory which contains the ``Chart.yaml``
file.
+ Make sure that you are inside the directory which contains the
``Chart.yaml`` file.
The chart will be downloaded and saved inside the ``charts/`` directory.
diff --git a/chart/docs/index.rst b/chart/docs/index.rst
index bd7c6305dd3..ae8fc85a2b3 100644
--- a/chart/docs/index.rst
+++ b/chart/docs/index.rst
@@ -62,25 +62,24 @@ deployment on a `Kubernetes <http://kubernetes.io>`__
cluster using the
Requirements
------------
-- Kubernetes 1.30+ cluster
-- Helm 3.10+
-- PV provisioner support in the underlying infrastructure (optionally)
+.. jinja:: global_ctx
+
+ - Kubernetes {{ min_k8s_version }}+ cluster
+ - Helm {{ helm_version }}+
+ - PV provisioner support in the underlying infrastructure (optionally)
Features
--------
* Supported executors: ``LocalExecutor``, ``CeleryExecutor``,
``KubernetesExecutor``
* Supported multiple Executors
-* Supported AWS executors with AWS provider version ``8.21.0+``:
+* Supported AWS executors with AWS provider version ``9.18.0+``:
* ``airflow.providers.amazon.aws.executors.batch.AwsBatchExecutor``
* ``airflow.providers.amazon.aws.executors.ecs.AwsEcsExecutor``
-
-* Supported AWS executors with AWS provider version ``9.9.0+``:
-
*
``airflow.providers.amazon.aws.executors.aws_lambda.lambda_executor.AwsLambdaExecutor``
-* Supported Edge executor with edge3 provider version ``1.0.0+``:
+* Supported Edge executor with edge3 provider version ``1.6.0+``:
* ``airflow.providers.edge3.executors.EdgeExecutor``
@@ -190,6 +189,10 @@ This will run database migrations every time there is a
``Sync`` event in Argo C
If you use the ``CeleryExecutor`` with the built-in Redis, it is recommended
that you set up a static Redis password either by supplying
``redis.passwordSecretName`` and ``data.brokerUrlSecretName`` or
``redis.password``.
+.. note::
+
+ Due to security concerns, it is not advised to set sensitive values like
passwords within the ``values.yaml``.
+
.. note::
By default, Helm hooks are also enabled for ``extraSecrets`` or
``extraConfigMaps``. When using the above CI/CD tools, you might encounter
issues due to these default hooks.
@@ -224,6 +227,10 @@ It is not enabled by default as this may cause unexpected
behaviours on existing
For existing installations, all your resources will be recreated with a new
name and helm will delete previous resources.
+.. note::
+
+ In current ``useStandardNaming`` implementation, the standard name is not
affecting Kubernetes Service Accounts created by Airflow Helm Chart.
+
This won't delete existing PVCs for logs used by StatefulSets/Deployments, but
it will recreate them with brand new PVCs.
If you do want to preserve logs history you'll need to manually copy the data
of these volumes into the new volumes after
deployment. Depending on what storage backend/class you're using, this
procedure may vary. If you don't mind starting
diff --git a/chart/docs/parameters-ref.rst b/chart/docs/parameters-ref.rst
index 521bcd2abce..49fd609f839 100644
--- a/chart/docs/parameters-ref.rst
+++ b/chart/docs/parameters-ref.rst
@@ -38,19 +38,21 @@ The following tables lists the configurable parameters of
the Airflow chart and
- Default
{% for param in section["params"] %}
+ {% if "deprecated" not in param["description"] %}
+
* - ``{{ param["name"] }}``
- {{ param["description"] }}
- ``{{ param["default"] }}``
- {% if param["examples"] %}
+ {% if param["examples"] %}
.. code-block:: yaml
:caption: Examples
{{ param["examples"] | indent(width=10) }}
+ {% endif %}
{% endif %}
{% endfor %}
-
{% endfor %}
@@ -75,3 +77,37 @@ and install the chart:
.. code-block:: bash
helm install my-release apache-airflow/airflow --values override-values.yaml
+
+Deprecated Parameters
+=====================
+
+The following table contains all deprecated configuration parameters of the
Airflow chart with their default values. All values defined in below table with
be removed
+in the next Helm Chart major release.
+
+.. jinja:: params_ctx
+
+ .. list-table::
+ :widths: 15 10 30
+ :header-rows: 1
+
+ * - Parameter
+ - Description
+ - Default
+
+ {% for section in sections %}
+ {% for param in section["params"] %}
+ {% if "deprecated" in param["description"] %}
+ * - ``{{ param["name"] }}``
+ - {{ param["description"] }}
+ - ``{{ param["default"] }}``
+ {% if param["examples"] %}
+
+ .. code-block:: yaml
+ :caption: Examples
+
+ {{ param["examples"] | indent(width=10) }}
+
+ {% endif %}
+ {% endif %}
+ {% endfor %}
+ {% endfor %}
diff --git a/chart/docs/production-guide.rst b/chart/docs/production-guide.rst
index ea8bbc88353..3e5ffc51da6 100644
--- a/chart/docs/production-guide.rst
+++ b/chart/docs/production-guide.rst
@@ -280,14 +280,19 @@ This setting can be configured in the Airflow chart at
different levels:
:caption: values.yaml
workers:
- safeToEvict: true
+ celery:
+ safeToEvict: true
+ kubernetes:
+ safeToEvict: true
scheduler:
safeToEvict: true
apiServer:
safeToEvict: true
-``workers.safeToEvict`` defaults to ``false``, and when using
``KubernetesExecutor``
-``workers.safeToEvict`` shouldn't be set to ``true`` as the workers may be
removed before finishing.
+.. note::
+
+ ``workers.kubernetes.safeToEvict`` defaults to ``false`` as it shouldn't be
set to ``true``,
+ because the tasks may be removed before finishing.
Extending and customizing Airflow Image
---------------------------------------
diff --git a/chart/docs/quick-start.rst b/chart/docs/quick-start.rst
index 399e47d4900..12c038185f6 100644
--- a/chart/docs/quick-start.rst
+++ b/chart/docs/quick-start.rst
@@ -26,11 +26,14 @@ Run Airflow on Kind cluster
Install kind and create a cluster
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-We recommend testing with Kubernetes 1.30+, example:
-.. code-block:: bash
+.. jinja:: global_ctx
+
+ We recommend testing with Kubernetes {{ min_k8s_version }}+, example:
+
+ .. code-block:: bash
- kind create cluster --image kindest/node:v1.30.13
+ kind create cluster --image kindest/node:{{ min_k8s_version }}
Confirm it's up:
diff --git a/chart/docs/setting-resources-for-containers.rst
b/chart/docs/setting-resources-for-containers.rst
index 31cdbb6adf5..fae0e32fbe4 100644
--- a/chart/docs/setting-resources-for-containers.rst
+++ b/chart/docs/setting-resources-for-containers.rst
@@ -28,47 +28,26 @@ You can define different resources for various Airflow
containers. By default th
Possible containers where resources can be configured include:
-* Main Airflow containers and their sidecars. You can add the resources for
these containers through the following parameters:
+.. jinja:: params_ctx
- * ``workers.resources``
- * ``workers.celery.logGroomerSidecar.resources``
- * ``workers.kerberosSidecar.resources``
- * ``workers.kerberosInitContainer.resources``
- * ``scheduler.resources``
- * ``scheduler.logGroomerSidecar.resources``
- * ``dags.gitSync.resources``
- * ``apiServer.resources``
- * ``flower.resources``
- * ``dagProcessor.resources``
- * ``dagProcessor.logGroomerSidecar.resources``
- * ``triggerer.resources``
- * ``triggerer.logGroomerSidecar.resources``
+ {% for section in sections %}
+ {% for param in section["params"] %}
+ {% if param["name"].endswith("resources") %}
+ * ``{{ param["name"] }}``
+ {% endif %}
+ {% endfor %}
+ {% endfor %}
-* Containers used for Airflow Kubernetes jobs or cron jobs. You can add the
resources for these containers through the following parameters:
-
- * ``cleanup.resources``
- * ``createUserJob.resources``
- * ``migrateDatabaseJob.resources``
- * ``databaseCleanup.resources``
-
-* Other containers that can be deployed by the chart. You can add the
resources for these containers through the following parameters:
-
- * ``statsd.resources``
- * ``pgbouncer.resources``
- * ``pgbouncer.metricsExporterSidecar.resources``
- * ``redis.resources``
-
-For example, specifying resources for worker Kerberos sidecar:
+For example, specifying resources for scheduler container:
.. code-block:: yaml
:caption: values.yaml
- workers:
- kerberosSidecar:
- resources:
- limits:
- cpu: 200m
- memory: 256Mi
- requests:
- cpu: 100m
- memory: 128Mi
+ scheduler:
+ resources:
+ limits:
+ cpu: 1
+ memory: 1Gi
+ requests:
+ cpu: 500m
+ memory: 512Gi
diff --git a/chart/docs/upgrading-to-airflow-3.rst
b/chart/docs/upgrading-to-airflow-3.rst
index 312bceb8028..f3555da0ef3 100644
--- a/chart/docs/upgrading-to-airflow-3.rst
+++ b/chart/docs/upgrading-to-airflow-3.rst
@@ -22,22 +22,20 @@ Before reading this, make sure you have read the Airflow
Upgrade Guide for how t
:doc:`apache-airflow:installation/upgrading_to_airflow3`.
This guide describes the chart-specific tasks for upgrading a deployment of
the official Airflow Helm chart from
-Airflow 2.x to Airflow 3.x. Upgrade the chart to ``1.16.0`` or later (chart
``1.16.0`` is the first release that
-supports Airflow 3) and review your ``values.yaml`` against the items below.
Many keys have been renamed, removed,
-or replaced and the chart will fail to render or deploy unsupported components
silently if old keys remain.
+Airflow 2.x to Airflow 3.x. Upgrade the chart to at least ``1.19.0`` version
and review your ``values.yaml`` against
+the items below. Many keys have been renamed, removed, or replaced within the
chart and the chart may fail to render or
+deploy unsupported components silently if old keys remain.
Bump the Airflow image
----------------------
-Set ``defaultAirflowTag`` and ``airflowVersion`` to a 3.x release. The chart
ships with a 3.x default starting at
-chart ``1.17.0``.
+Set ``defaultAirflowTag`` and ``airflowVersion`` to a 3.x release.
Rename ``webserver`` to ``apiServer``
-------------------------------------
The Airflow 3 component that serves the UI and the public REST API is the API
server, not the webserver. All
-configuration that lived under ``webserver:`` in the chart values must move
under ``apiServer:``. The component now
-listens on port ``8080`` and is started with ``airflow api-server``. For
example:
+configuration that lived under ``webserver:`` in the chart values must move
under ``apiServer:``. For example:
.. code-block:: yaml
@@ -65,17 +63,16 @@ Add a JWT secret
----------------
Airflow 3 uses short-lived JWT tokens to authenticate workers and triggerers
against the API server. The chart
-generates a ``jwt-secret`` Secret on install. If you template Secrets
out-of-band or pin a custom name through
-``jwtSecretName``, make sure the referenced Secret exists with a
``jwt-secret`` key before workers and triggerers
-start.
+generates a ``jwt-secret`` Secret on every install/update operation. If you
template Secrets out-of-band or
+pin a custom name through ``jwtSecretName``, make sure the referenced Secret
exists with a ``jwt-secret`` key
+before Airflow components start.
-Deploy the standalone Dag processor
+Deploy the standalone Dag Processor
-----------------------------------
-In Airflow 3 the Dag file processor is no longer embedded in the scheduler.
The chart deploys it as a separate
+In Airflow 3 the Dag Processor is no longer embedded in the scheduler. The
chart deploys it as a separate
``dag-processor`` Deployment configured under ``dagProcessor:``. Review
resources, tolerations, and any RBAC
-overrides for this new component, including granting its ServiceAccount access
to your Dag bundles (git-sync,
-persistent volumes, or custom bundles).
+overrides for this component, including granting its Service Account access to
your Dags code.
Pick an auth manager
--------------------
@@ -97,8 +94,7 @@ for details.
Check the minimum Kubernetes version
------------------------------------
-Chart ``1.17.0`` raised the minimum supported Kubernetes version to ``1.30``.
Upgrade your cluster before upgrading
-the chart if needed.
+Chart ``1.19.0`` supports ``1.30+`` Kubernetes version. Upgrade your cluster
before upgrading the chart if needed.
Run database migrations as part of the upgrade
----------------------------------------------
@@ -112,7 +108,7 @@ Re-check renamed or removed values
----------------------------------
Many configuration options under ``webserver``/``apiServer``, ``workers``, and
``scheduler`` were renamed or
-removed across chart ``1.16.0``..``1.18.0``. Diff your existing
``values.yaml`` against the chart's default
+removed across chart versions. Diff your existing ``values.yaml`` against the
chart's default
``values.yaml`` and the :doc:`release_notes` for those versions before
applying.
After updating ``values.yaml``, render the chart locally with ``helm
template`` and inspect the diff against your
diff --git a/chart/newsfragments/66970.significant.rst
b/chart/newsfragments/66970.significant.rst
new file mode 100644
index 00000000000..e72b0d10220
--- /dev/null
+++ b/chart/newsfragments/66970.significant.rst
@@ -0,0 +1 @@
+Minimum Helm version was updated to ``3.19.0``
