This is an automated email from the ASF dual-hosted git repository.
jscheffl pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new 17c597e3e5e `docs/dag-bundle-docs`: Adding more specific docs for
configuring DAG bundles (#67843)
17c597e3e5e is described below
commit 17c597e3e5e6e2dd2080f5128a72f5451f2c6e77
Author: Jake McGrath <[email protected]>
AuthorDate: Mon Jun 1 16:49:06 2026 -0400
`docs/dag-bundle-docs`: Adding more specific docs for configuring DAG
bundles (#67843)
* docs/dag-bundle-docs: Adding more specific docs for configuring DAG
bundles
* docs/dag-bundle-docs: Renaming, linking other docs
* Update airflow-core/docs/administration-and-deployment/dag-bundles.rst
Co-authored-by: Jens Scheffler <[email protected]>
---------
Co-authored-by: Jens Scheffler <[email protected]>
---
.../administration-and-deployment/dag-bundles.rst | 94 +++++++++++++++++++++-
1 file changed, 90 insertions(+), 4 deletions(-)
diff --git a/airflow-core/docs/administration-and-deployment/dag-bundles.rst
b/airflow-core/docs/administration-and-deployment/dag-bundles.rst
index 74cf2ed298d..7e1eaf0123b 100644
--- a/airflow-core/docs/administration-and-deployment/dag-bundles.rst
+++ b/airflow-core/docs/administration-and-deployment/dag-bundles.rst
@@ -48,7 +48,7 @@ Airflow supports multiple types of Dag Bundles, each catering
to specific use ca
These bundles reference a local directory containing Dag files. They are
ideal for development and testing environments, but do not support versioning
of the bundle, meaning tasks always run using the latest code.
**airflow.providers.git.bundles.git.GitDagBundle**
- These bundles integrate with Git repositories, allowing Airflow to fetch
Dags directly from a repository.
+ These bundles integrate with Git repositories, allowing Airflow to fetch
Dags directly from a repository. The `GitDagBundle` does support versioning.
**airflow.providers.amazon.aws.bundles.s3.S3DagBundle**
These bundles reference an S3 bucket containing Dag files. They do not
support versioning of the bundle, meaning tasks always run using the latest
code.
@@ -61,16 +61,102 @@ Configuring Dag bundles
Dag bundles are configured in
:ref:`config:dag_processor__dag_bundle_config_list`. You can add one or more
Dag bundles here.
-By default, Airflow adds a local Dag bundle, which is the same as the old Dags
folder. This is done for backwards compatibility, and you can remove it if you
do not want to use it. You can also keep it and add other Dag bundles, such as
a git Dag bundle.
+By default, Airflow adds a ``LocalDagBundle`` pointing at the configured Dags
folder, maintaining the same behaviour as Airflow 2's Dags folder. The only
kwarg is ``path``, which defaults to the value of
:ref:`config:core__dags_folder` when omitted:
-For example, adding multiple Dag bundles to your ``airflow.cfg`` file:
+.. code-block:: ini
+
+ [dag_processor]
+ dag_bundle_config_list = [
+ {
+ "name": "dags-folder",
+ "classpath": "airflow.dag_processing.bundles.local.LocalDagBundle",
+ "kwargs": {
+ "path": "/opt/airflow/dags"
+ }
+ }
+ ]
+
+.. note::
+
+ ``LocalDagBundle`` does not support versioning. Tasks always run against
the latest code on disk.
+
+For a Git Dag bundle, the only required kwarg is ``tracking_ref`` (a branch,
tag, or commit SHA). Use ``git_conn_id`` to reference an Airflow connection
that holds the repository credentials, or supply ``repo_url`` directly. You can
also narrow the checkout to a subdirectory with ``subdir``, or use
``sparse_dirs`` to enable a sparse checkout of specific directories:
.. code-block:: ini
[dag_processor]
dag_bundle_config_list = [
{
- "name": "my_git_repo",
+ "name": "my-git-repo",
+ "classpath": "airflow.providers.git.bundles.git.GitDagBundle",
+ "kwargs": {
+ "git_conn_id": "my_git_conn",
+ "subdir": "dags",
+ "tracking_ref": "main",
+ }
+ }
+ ]
+
+.. note::
+
+ ``GitDagBundle`` supports versioning. Each Dag run records the Git commit
it was created with, allowing reruns to use the exact same code even if the
repository has since been updated.
+
+See :doc:`apache-airflow-providers-git:bundles/index` for the full list of
kwargs and more examples.
+
+For an S3 Dag bundle, the required kwarg is ``bucket_name``. You can
optionally set ``aws_conn_id`` (defaults to ``aws_default``) and ``prefix`` to
scope the bundle to a subdirectory within the bucket:
+
+.. code-block:: ini
+
+ [dag_processor]
+ dag_bundle_config_list = [
+ {
+ "name": "my-s3-dags",
+ "classpath": "airflow.providers.amazon.aws.bundles.s3.S3DagBundle",
+ "kwargs": {
+ "aws_conn_id": "aws_default",
+ "bucket_name": "my-airflow-bucket",
+ "prefix": "dags/"
+ }
+ }
+ ]
+
+.. note::
+
+ ``S3DagBundle`` does not support versioning. Tasks always run against the
latest code in the bucket.
+
+See :doc:`apache-airflow-providers-amazon:bundles/index` for the full list of
kwargs and more examples.
+
+For a GCS Dag bundle, the required kwarg is ``bucket_name``. You can
optionally set ``gcp_conn_id`` (defaults to ``google_cloud_default``) and
``prefix`` to scope the bundle to a subdirectory within the bucket:
+
+.. code-block:: ini
+
+ [dag_processor]
+ dag_bundle_config_list = [
+ {
+ "name": "my-gcs-dags",
+ "classpath":
"airflow.providers.google.cloud.bundles.gcs.GCSDagBundle",
+ "kwargs": {
+ "gcp_conn_id": "google_cloud_default",
+ "bucket_name": "my-airflow-bucket",
+ "prefix": "dags/"
+ }
+ }
+ ]
+
+.. note::
+
+ ``GCSDagBundle`` does not support versioning. Tasks always run against the
latest code in the bucket.
+
+See :doc:`apache-airflow-providers-google:bundles/index` for the full list of
kwargs and more examples.
+
+You can combine multiple bundle types in a single deployment. The default
``LocalDagBundle`` can be removed if you no longer need it, or kept alongside
other bundles:
+
+.. code-block:: ini
+
+ [dag_processor]
+ dag_bundle_config_list = [
+ {
+ "name": "my_git_bundle",
"classpath": "airflow.providers.git.bundles.git.GitDagBundle",
"kwargs": {"tracking_ref": "main", "git_conn_id": "my_git_conn"}
},
