kaxil commented on code in PR #74748: URL: https://github.com/apache/airflow/pull/74748#discussion_r4275228700
########## airflow-core/docs/authoring-and-scheduling/yaml-dags.rst: ########## @@ -0,0 +1,349 @@ + .. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +.. _yaml-dags: + +Writing Dags in YAML +==================== + +|experimental| + +.. versionadded:: 3.4.0 + +Airflow can build a Dag from a YAML document. One YAML document describes one +Dag, and the YAML Dag importer turns it into the same Dag structure that Airflow +runs like any other source. The tasks can be ordinary Airflow operators, so +every provider operator is available without writing code. Review Comment: Is "every provider operator" too strong? Arguments can only be YAML values, and `_build_operator` hands them straight to the constructor, so anything that needs a Python object can't be built. `PythonOperator` and `BranchPythonOperator` need a real callable for `python_callable`, and until #74749 lands any timedelta-only argument is out too. Maybe "any operator whose arguments can be written as YAML values". ########## airflow-core/docs/authoring-and-scheduling/yaml-dags.rst: ########## @@ -0,0 +1,349 @@ + .. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +.. _yaml-dags: + +Writing Dags in YAML +==================== + +|experimental| + +.. versionadded:: 3.4.0 + +Airflow can build a Dag from a YAML document. One YAML document describes one +Dag, and the YAML Dag importer turns it into the same Dag structure that Airflow +runs like any other source. The tasks can be ordinary Airflow operators, so +every provider operator is available without writing code. + +A document is declarative. Everything in it is a value, and the shape of each +value decides what it means. Since the format round-trips through JSON, a +document may equally be written as JSON. Review Comment: The importer only claims `.yaml`/`.yml` (`supported_extensions` in importer.py), so a `dag.json` is never discovered. A JSON body works inside a `.yaml` file, or with `.json` added to the importer's `extensions`. Could this say which of those it means? ########## airflow-core/docs/authoring-and-scheduling/yaml-dags.rst: ########## @@ -0,0 +1,349 @@ + .. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +.. _yaml-dags: + +Writing Dags in YAML +==================== + +|experimental| + +.. versionadded:: 3.4.0 + +Airflow can build a Dag from a YAML document. One YAML document describes one +Dag, and the YAML Dag importer turns it into the same Dag structure that Airflow +runs like any other source. The tasks can be ordinary Airflow operators, so +every provider operator is available without writing code. + +A document is declarative. Everything in it is a value, and the shape of each +value decides what it means. Since the format round-trips through JSON, a +document may equally be written as JSON. + +The following is a example YAML file defining a Dag: + +.. code-block:: yaml + + $schema: https://airflow.apache.org/schemas/dag/2026-10-30.json + dag_id: daily_report + schedule: "0 3 * * *" + start_date: 2024-01-01 + + tasks: + - id: extract + uses: airflow.providers.standard.operators.bash.BashOperator + with: + bash_command: + $t: "extract.sh {{ ds }}" + + - id: publish + uses: airflow.providers.standard.operators.bash.BashOperator + needs: [extract] + with: + bash_command: "publish.sh" + +This Dag runs every day at 03:00, The first task ``extract`` renders the date +into its command. The second ``publish`` waits for ``extract`` through +``needs`` and runs a fixed command. + +Enabling the importer +--------------------- + +The YAML importer is opt-in. Register it in +:ref:`config:dag_processor__dag_importer_configs`: + +.. code-block:: ini + + [dag_processor] + dag_importer_configs = [{"classpath": "airflow.sdk.importers.yaml_importer.YamlDagImporter"}] + +The importer handles ``.yaml`` and ``.yml`` files that declare a Dag ``$schema`` Review Comment: This is the snippet people will copy, so maybe one line here saying the same setting is needed on the workers (and wherever `airflow dags` runs). dag-importers.rst says so in an `important` box, but it's easy to miss from this page. ########## airflow-core/docs/authoring-and-scheduling/yaml-dags.rst: ########## @@ -0,0 +1,349 @@ + .. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +.. _yaml-dags: + +Writing Dags in YAML +==================== + +|experimental| + +.. versionadded:: 3.4.0 + +Airflow can build a Dag from a YAML document. One YAML document describes one +Dag, and the YAML Dag importer turns it into the same Dag structure that Airflow +runs like any other source. The tasks can be ordinary Airflow operators, so +every provider operator is available without writing code. + +A document is declarative. Everything in it is a value, and the shape of each +value decides what it means. Since the format round-trips through JSON, a +document may equally be written as JSON. + +The following is a example YAML file defining a Dag: Review Comment: A few typos on the page: 36 "a example", 58 "03:00, The", 88 "Attribute of the Dag sit", 165 "specifies" (two subjects), 185 "The contents ... is read", 234 "at the task-level", 261 "two things must satisfy", 264 "Try to use ... results in", 267 "A field leave unset", 282 "has be performed". ########## airflow-core/docs/authoring-and-scheduling/yaml-dags.rst: ########## @@ -0,0 +1,349 @@ + .. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +.. _yaml-dags: + +Writing Dags in YAML +==================== + +|experimental| + +.. versionadded:: 3.4.0 + +Airflow can build a Dag from a YAML document. One YAML document describes one +Dag, and the YAML Dag importer turns it into the same Dag structure that Airflow +runs like any other source. The tasks can be ordinary Airflow operators, so +every provider operator is available without writing code. + +A document is declarative. Everything in it is a value, and the shape of each +value decides what it means. Since the format round-trips through JSON, a +document may equally be written as JSON. + +The following is a example YAML file defining a Dag: + +.. code-block:: yaml + + $schema: https://airflow.apache.org/schemas/dag/2026-10-30.json + dag_id: daily_report + schedule: "0 3 * * *" + start_date: 2024-01-01 + + tasks: + - id: extract + uses: airflow.providers.standard.operators.bash.BashOperator + with: + bash_command: + $t: "extract.sh {{ ds }}" + + - id: publish + uses: airflow.providers.standard.operators.bash.BashOperator + needs: [extract] + with: + bash_command: "publish.sh" + +This Dag runs every day at 03:00, The first task ``extract`` renders the date +into its command. The second ``publish`` waits for ``extract`` through +``needs`` and runs a fixed command. + +Enabling the importer +--------------------- + +The YAML importer is opt-in. Register it in +:ref:`config:dag_processor__dag_importer_configs`: + +.. code-block:: ini + + [dag_processor] + dag_importer_configs = [{"classpath": "airflow.sdk.importers.yaml_importer.YamlDagImporter"}] + +The importer handles ``.yaml`` and ``.yml`` files that declare a Dag ``$schema`` +(see below). For per-bundle registration and other options, see +:doc:`../administration-and-deployment/dag-importers`. + +The shape of a document +----------------------- + +``$schema`` (required) + The format version the document is written against, given as the URL of a + published schema. The version it names selects the rules the document is + parsed with. ``$schema`` is also how Airflow recognizes the file + as a Dag; a file without it is silently skipped. Editors that understand + ``$schema`` will offer validation and completion with no extra setup. + +Dag-level keys + Attribute of the Dag sit at the top level, such as ``dag_id``, ``schedule``, + ``params``, etc. An unknown key is reported as an error. + +``tasks`` + A list of tasks. Each task has an ``id`` and exactly one body. + +Task body: ``uses`` or ``run`` + ``uses:`` names a ready-made operator by its fully-qualified Python import + path (e.g. ``airflow.providers.standard.operators.bash.BashOperator``). + ``run:`` holds the arguments of a code task you implement elsewhere (see + `Code tasks`_ below). A task may have one or the other. + +``with`` + Operator-specific arguments. These are what you would pass to the operator's + constructor. Read on to understand the value grammar you can use in these + arguments. + +Task-level keys + Scheduling and common attributes sit next to the body, distinct from + ``with:``. These include things like ``retries``, ``trigger_rule``, + ``queue``, etc. Any key that is not one of the structural keys (``id``, + ``uses``, ``run``, ``with``, ``needs``, ``extends``) is passed to the + operator type, which validates it. Review Comment: The importer checks these itself before the operator sees them: a task-level key that isn't a `BaseOperator` argument fails in `_build_operator` with "are not task-level (BaseOperator) arguments; operator-specific arguments belong under 'with'". So it isn't passed to the operator type. Maybe "must be a base operator argument; operator-specific ones go under `with:`". ########## airflow-core/docs/authoring-and-scheduling/yaml-dags.rst: ########## @@ -0,0 +1,349 @@ + .. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +.. _yaml-dags: + +Writing Dags in YAML +==================== + +|experimental| + +.. versionadded:: 3.4.0 + +Airflow can build a Dag from a YAML document. One YAML document describes one +Dag, and the YAML Dag importer turns it into the same Dag structure that Airflow +runs like any other source. The tasks can be ordinary Airflow operators, so +every provider operator is available without writing code. + +A document is declarative. Everything in it is a value, and the shape of each +value decides what it means. Since the format round-trips through JSON, a +document may equally be written as JSON. + +The following is a example YAML file defining a Dag: + +.. code-block:: yaml + + $schema: https://airflow.apache.org/schemas/dag/2026-10-30.json + dag_id: daily_report + schedule: "0 3 * * *" + start_date: 2024-01-01 + + tasks: + - id: extract + uses: airflow.providers.standard.operators.bash.BashOperator + with: + bash_command: + $t: "extract.sh {{ ds }}" + + - id: publish + uses: airflow.providers.standard.operators.bash.BashOperator + needs: [extract] + with: + bash_command: "publish.sh" + +This Dag runs every day at 03:00, The first task ``extract`` renders the date +into its command. The second ``publish`` waits for ``extract`` through +``needs`` and runs a fixed command. + +Enabling the importer +--------------------- + +The YAML importer is opt-in. Register it in +:ref:`config:dag_processor__dag_importer_configs`: + +.. code-block:: ini + + [dag_processor] + dag_importer_configs = [{"classpath": "airflow.sdk.importers.yaml_importer.YamlDagImporter"}] + +The importer handles ``.yaml`` and ``.yml`` files that declare a Dag ``$schema`` +(see below). For per-bundle registration and other options, see +:doc:`../administration-and-deployment/dag-importers`. + +The shape of a document +----------------------- + +``$schema`` (required) + The format version the document is written against, given as the URL of a + published schema. The version it names selects the rules the document is + parsed with. ``$schema`` is also how Airflow recognizes the file + as a Dag; a file without it is silently skipped. Editors that understand + ``$schema`` will offer validation and completion with no extra setup. + +Dag-level keys + Attribute of the Dag sit at the top level, such as ``dag_id``, ``schedule``, + ``params``, etc. An unknown key is reported as an error. + +``tasks`` + A list of tasks. Each task has an ``id`` and exactly one body. Review Comment: Not quite "exactly one": a task with neither `uses` nor `run` builds an `EmptyOperator` (the `task.uses is None` branch in `_build_operator`, and test_importer.py covers `{id: a}`). "At most one body; a task with neither is an `EmptyOperator`" would match the code, and it's useful for users to know. ########## airflow-core/docs/authoring-and-scheduling/yaml-dags.rst: ########## @@ -0,0 +1,349 @@ + .. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +.. _yaml-dags: + +Writing Dags in YAML +==================== + +|experimental| + +.. versionadded:: 3.4.0 + +Airflow can build a Dag from a YAML document. One YAML document describes one +Dag, and the YAML Dag importer turns it into the same Dag structure that Airflow +runs like any other source. The tasks can be ordinary Airflow operators, so +every provider operator is available without writing code. + +A document is declarative. Everything in it is a value, and the shape of each +value decides what it means. Since the format round-trips through JSON, a +document may equally be written as JSON. + +The following is a example YAML file defining a Dag: + +.. code-block:: yaml + + $schema: https://airflow.apache.org/schemas/dag/2026-10-30.json + dag_id: daily_report + schedule: "0 3 * * *" + start_date: 2024-01-01 + + tasks: + - id: extract + uses: airflow.providers.standard.operators.bash.BashOperator + with: + bash_command: + $t: "extract.sh {{ ds }}" + + - id: publish + uses: airflow.providers.standard.operators.bash.BashOperator + needs: [extract] + with: + bash_command: "publish.sh" + +This Dag runs every day at 03:00, The first task ``extract`` renders the date +into its command. The second ``publish`` waits for ``extract`` through +``needs`` and runs a fixed command. + +Enabling the importer +--------------------- + +The YAML importer is opt-in. Register it in +:ref:`config:dag_processor__dag_importer_configs`: + +.. code-block:: ini + + [dag_processor] + dag_importer_configs = [{"classpath": "airflow.sdk.importers.yaml_importer.YamlDagImporter"}] + +The importer handles ``.yaml`` and ``.yml`` files that declare a Dag ``$schema`` +(see below). For per-bundle registration and other options, see +:doc:`../administration-and-deployment/dag-importers`. + +The shape of a document +----------------------- + +``$schema`` (required) + The format version the document is written against, given as the URL of a + published schema. The version it names selects the rules the document is + parsed with. ``$schema`` is also how Airflow recognizes the file + as a Dag; a file without it is silently skipped. Editors that understand + ``$schema`` will offer validation and completion with no extra setup. + +Dag-level keys + Attribute of the Dag sit at the top level, such as ``dag_id``, ``schedule``, + ``params``, etc. An unknown key is reported as an error. + +``tasks`` + A list of tasks. Each task has an ``id`` and exactly one body. + +Task body: ``uses`` or ``run`` + ``uses:`` names a ready-made operator by its fully-qualified Python import + path (e.g. ``airflow.providers.standard.operators.bash.BashOperator``). + ``run:`` holds the arguments of a code task you implement elsewhere (see + `Code tasks`_ below). A task may have one or the other. + +``with`` + Operator-specific arguments. These are what you would pass to the operator's + constructor. Read on to understand the value grammar you can use in these + arguments. + +Task-level keys + Scheduling and common attributes sit next to the body, distinct from + ``with:``. These include things like ``retries``, ``trigger_rule``, + ``queue``, etc. Any key that is not one of the structural keys (``id``, + ``uses``, ``run``, ``with``, ``needs``, ``extends``) is passed to the + operator type, which validates it. + +``needs`` + A list of upstream tasks (referenced by ``id``) that must run before this + one. This describes an explicit dependency edge; XCom references in + ``with:`` or ``run:`` are discovered automatically and do not need to be + repeated here. + + +Templates and extends +--------------------- + +Tasks often share arguments. You can define a named partial task under the +top-level ``templates:`` key, and pull it into a task with ``extends``: + +.. code-block:: yaml + + templates: + retryable: + retries: 2 + shell: + extends: [retryable] # a template can extend other templates + uses: airflow.providers.standard.operators.bash.BashOperator + with: + cwd: /opt/airflow/scripts + + tasks: + - id: extract + extends: [shell] # pull the template into this task + with: + bash_command: "extract.sh" + - id: load + extends: [shell] + with: + bash_command: "load.sh" + +Both tasks run a ``BashOperator`` with ``retries: 2`` and the shared ``cwd``, +and add only their own ``bash_command``. A template may set anything a task can, +and may itself ``extends`` other templates. + +The templates named in ``extends`` are applied left to right, and the task's own +inline fields are applied last. Merging is shallow: ``with:`` and ``run:`` are +combined one key at a time, while every other field is taken as a whole rather +than deep-merged. + +The assembled task is validated against the real operator, so a field the +operator does not accept is reported as an error. It is also an error if a +non-existent template is referenced. + + +Values are literal by default +----------------------------- + +A value under ``with:`` (or ``run:``) is taken literally, unless it is a +single-key object whose key is a reserved ``$``-marker. For example, +``"echo foo.sh"`` and ``"{{ not rendered }}"`` specifies exactly that string, +not a template, even if ``bash_command`` is listed on the operator as a template +field. + +Four markers opt a value into special handling: + +``{$t: "..."}`` is a template string + A Jinja template, rendered when the task runs. This is the only way a value + is templated: ``{$t: "{{ ds }}"}``, ``{$t: "{{ params.region }}"}``, + ``{$t: "{{ conn.my_db.host }}"}``. The long form ``{$template: "..."}`` is + also accepted. + +``{$x: id}`` references an upstream output + The XCom output of an upstream task. ``{$x: extract}`` references + ``extract``'s return value, and ``{$x: {task: extract, key: rows}}`` a + specific key. Binding a value this way also creates the dependency edge + implicitly, so a task that consumes another's output does not need to also + list it in ``needs:``. The long form is ``{$xcom: ...}``. + +``{$f: path}`` reads a file's contents + The contents of a file stored next to the document is read and used. + ``{$t: {$f: path}}`` allows reading the file and then rendering its contents + as a template. The long form is ``{$file: path}``. A file is loaded only + when you ask with ``$f``; it is never inferred from a filename suffix. + +``{$const: value}`` is a forced literal + An escape for the rare case that real data is shaped exactly like a marker. + ``{$const: {$x: 1}}`` is the literal dict ``{"$x": 1}``, not an XCom. + +Markers nest. A marker inside a list, or inside a nested mapping such as an +``env`` dict, is resolved by the same rules, so references and templates compose +through structure: + +.. code-block:: yaml + + with: + env: + RUN_DATE: + $t: "{{ ds }}" # rendered + REGION: us-east-1 # literal + +Dates and datetimes +------------------- + +A date or datetime argument accepts an ISO-8601 string or a YAML-native date, +and they mean the same thing: + +.. code-block:: yaml + + start_date: "2024-01-01" # ISO-8601 string + start_date: 2024-01-01 # YAML date Review Comment: This block repeats `start_date`, and the parser rejects duplicate keys, so copying it fails with "found duplicate key 'start_date'". Maybe comment one of them out with `# or:`. ########## airflow-core/docs/authoring-and-scheduling/yaml-dags.rst: ########## @@ -0,0 +1,349 @@ + .. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +.. _yaml-dags: + +Writing Dags in YAML +==================== + +|experimental| + +.. versionadded:: 3.4.0 + +Airflow can build a Dag from a YAML document. One YAML document describes one +Dag, and the YAML Dag importer turns it into the same Dag structure that Airflow +runs like any other source. The tasks can be ordinary Airflow operators, so +every provider operator is available without writing code. + +A document is declarative. Everything in it is a value, and the shape of each +value decides what it means. Since the format round-trips through JSON, a +document may equally be written as JSON. + +The following is a example YAML file defining a Dag: + +.. code-block:: yaml + + $schema: https://airflow.apache.org/schemas/dag/2026-10-30.json + dag_id: daily_report + schedule: "0 3 * * *" + start_date: 2024-01-01 + + tasks: + - id: extract + uses: airflow.providers.standard.operators.bash.BashOperator + with: + bash_command: + $t: "extract.sh {{ ds }}" + + - id: publish + uses: airflow.providers.standard.operators.bash.BashOperator + needs: [extract] + with: + bash_command: "publish.sh" + +This Dag runs every day at 03:00, The first task ``extract`` renders the date +into its command. The second ``publish`` waits for ``extract`` through +``needs`` and runs a fixed command. + +Enabling the importer +--------------------- + +The YAML importer is opt-in. Register it in +:ref:`config:dag_processor__dag_importer_configs`: + +.. code-block:: ini + + [dag_processor] + dag_importer_configs = [{"classpath": "airflow.sdk.importers.yaml_importer.YamlDagImporter"}] + +The importer handles ``.yaml`` and ``.yml`` files that declare a Dag ``$schema`` +(see below). For per-bundle registration and other options, see +:doc:`../administration-and-deployment/dag-importers`. + +The shape of a document +----------------------- + +``$schema`` (required) + The format version the document is written against, given as the URL of a + published schema. The version it names selects the rules the document is + parsed with. ``$schema`` is also how Airflow recognizes the file + as a Dag; a file without it is silently skipped. Editors that understand + ``$schema`` will offer validation and completion with no extra setup. + +Dag-level keys + Attribute of the Dag sit at the top level, such as ``dag_id``, ``schedule``, + ``params``, etc. An unknown key is reported as an error. + +``tasks`` + A list of tasks. Each task has an ``id`` and exactly one body. + +Task body: ``uses`` or ``run`` + ``uses:`` names a ready-made operator by its fully-qualified Python import + path (e.g. ``airflow.providers.standard.operators.bash.BashOperator``). + ``run:`` holds the arguments of a code task you implement elsewhere (see + `Code tasks`_ below). A task may have one or the other. + +``with`` + Operator-specific arguments. These are what you would pass to the operator's + constructor. Read on to understand the value grammar you can use in these + arguments. + +Task-level keys + Scheduling and common attributes sit next to the body, distinct from + ``with:``. These include things like ``retries``, ``trigger_rule``, + ``queue``, etc. Any key that is not one of the structural keys (``id``, + ``uses``, ``run``, ``with``, ``needs``, ``extends``) is passed to the + operator type, which validates it. + +``needs`` + A list of upstream tasks (referenced by ``id``) that must run before this + one. This describes an explicit dependency edge; XCom references in + ``with:`` or ``run:`` are discovered automatically and do not need to be + repeated here. + + +Templates and extends +--------------------- + +Tasks often share arguments. You can define a named partial task under the +top-level ``templates:`` key, and pull it into a task with ``extends``: + +.. code-block:: yaml + + templates: + retryable: + retries: 2 + shell: + extends: [retryable] # a template can extend other templates + uses: airflow.providers.standard.operators.bash.BashOperator + with: + cwd: /opt/airflow/scripts + + tasks: + - id: extract + extends: [shell] # pull the template into this task + with: + bash_command: "extract.sh" + - id: load + extends: [shell] + with: + bash_command: "load.sh" + +Both tasks run a ``BashOperator`` with ``retries: 2`` and the shared ``cwd``, +and add only their own ``bash_command``. A template may set anything a task can, +and may itself ``extends`` other templates. + +The templates named in ``extends`` are applied left to right, and the task's own +inline fields are applied last. Merging is shallow: ``with:`` and ``run:`` are +combined one key at a time, while every other field is taken as a whole rather +than deep-merged. + +The assembled task is validated against the real operator, so a field the +operator does not accept is reported as an error. It is also an error if a +non-existent template is referenced. + + +Values are literal by default +----------------------------- + +A value under ``with:`` (or ``run:``) is taken literally, unless it is a +single-key object whose key is a reserved ``$``-marker. For example, +``"echo foo.sh"`` and ``"{{ not rendered }}"`` specifies exactly that string, +not a template, even if ``bash_command`` is listed on the operator as a template +field. + +Four markers opt a value into special handling: + +``{$t: "..."}`` is a template string + A Jinja template, rendered when the task runs. This is the only way a value + is templated: ``{$t: "{{ ds }}"}``, ``{$t: "{{ params.region }}"}``, + ``{$t: "{{ conn.my_db.host }}"}``. The long form ``{$template: "..."}`` is + also accepted. + +``{$x: id}`` references an upstream output + The XCom output of an upstream task. ``{$x: extract}`` references + ``extract``'s return value, and ``{$x: {task: extract, key: rows}}`` a + specific key. Binding a value this way also creates the dependency edge + implicitly, so a task that consumes another's output does not need to also + list it in ``needs:``. The long form is ``{$xcom: ...}``. + +``{$f: path}`` reads a file's contents + The contents of a file stored next to the document is read and used. + ``{$t: {$f: path}}`` allows reading the file and then rendering its contents + as a template. The long form is ``{$file: path}``. A file is loaded only + when you ask with ``$f``; it is never inferred from a filename suffix. + +``{$const: value}`` is a forced literal + An escape for the rare case that real data is shaped exactly like a marker. + ``{$const: {$x: 1}}`` is the literal dict ``{"$x": 1}``, not an XCom. + +Markers nest. A marker inside a list, or inside a nested mapping such as an +``env`` dict, is resolved by the same rules, so references and templates compose +through structure: + +.. code-block:: yaml + + with: + env: + RUN_DATE: + $t: "{{ ds }}" # rendered + REGION: us-east-1 # literal + +Dates and datetimes +------------------- + +A date or datetime argument accepts an ISO-8601 string or a YAML-native date, +and they mean the same thing: + +.. code-block:: yaml + + start_date: "2024-01-01" # ISO-8601 string + start_date: 2024-01-01 # YAML date + end_date: "2024-06-01T12:00:00" # ISO-8601 datetime string + +These are converted to a Python ``datetime`` before the Dag or task is built. A bare date becomes +midnight, and an ISO-8601 string is parsed. + +The conversion is applied only to an argument that expects a datetime. This is +decided from the operator's or Dag's argument **type annotation**. + +Using your own operators +------------------------ + +``uses:`` takes any operator's fully-qualified Python import path, so your own +operator class can work exactly like a provider's. + +.. code-block:: yaml + + - id: ingest + uses: my_company.operators.IngestOperator + trigger_rule: all_done # base arguments at the task-level + with: # your operator's own arguments + source: s3://bucket/incoming + cutoff: 2024-01-01 + +An operator takes two kinds of argument, kept in separate places. Your operator's own constructor +arguments go under ``with:``. The arguments every operator inherits from the base operator -- +``retries``, ``retry_delay``, ``trigger_rule``, ``queue``, ``pool``, ``priority_weight``, +``depends_on_past``, ``start_date``, and the rest of the task model -- go at the task level, beside +the body. Placing a base argument under ``with:``, or an operator-specific one at the task level, is +an error, so each argument has a single home. + +The task is constructed roughly like this: + +.. code-block:: python + + IngestOperator( + task_id="ingest", + source="s3://bucket/incoming", + cutoff=datetime.datetime(2024, 1, 1, 0, 0), + trigger_rule="all_done", + ) + +Templated argument rendering +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +A task argument is literal by default. For one to be rendered as a Jinja +template, two things must satisfy: the operator must allow that field to be +templated (for Python operators, by specifying it in ``template_fields``), and +the Dag author must use the ``{$t: ...}`` syntax. A value not marked explicitly +stays literal. Try to use the template syntax on a field not specified by the +operator results in an error. + +A field leave unset keeps the operator's own default, including any templating +the operator applies to that default. A value is also never read from a file Review Comment: This undersells it a bit. `_restrict_template_fields` keeps every templated field the author did not set in `with:`, and that includes fields the operator builds in `__init__` from your literal args. `SQLTableCheckOperator` builds `self.sql` from `checks`, so an unmarked `check_statement: "{{ params.x }}"` does get rendered through `sql`. Keeping those fields templated is right (they are what makes operators like `SQLIntervalCheckOperator` work), but the page should say so, since line 263 promises an unmarked value stays literal. ########## airflow-core/docs/authoring-and-scheduling/yaml-dags.rst: ########## @@ -0,0 +1,349 @@ + .. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +.. _yaml-dags: + +Writing Dags in YAML +==================== + +|experimental| + +.. versionadded:: 3.4.0 + +Airflow can build a Dag from a YAML document. One YAML document describes one +Dag, and the YAML Dag importer turns it into the same Dag structure that Airflow +runs like any other source. The tasks can be ordinary Airflow operators, so +every provider operator is available without writing code. + +A document is declarative. Everything in it is a value, and the shape of each +value decides what it means. Since the format round-trips through JSON, a +document may equally be written as JSON. + +The following is a example YAML file defining a Dag: + +.. code-block:: yaml + + $schema: https://airflow.apache.org/schemas/dag/2026-10-30.json + dag_id: daily_report + schedule: "0 3 * * *" + start_date: 2024-01-01 + + tasks: + - id: extract + uses: airflow.providers.standard.operators.bash.BashOperator + with: + bash_command: + $t: "extract.sh {{ ds }}" + + - id: publish + uses: airflow.providers.standard.operators.bash.BashOperator + needs: [extract] + with: + bash_command: "publish.sh" + +This Dag runs every day at 03:00, The first task ``extract`` renders the date +into its command. The second ``publish`` waits for ``extract`` through +``needs`` and runs a fixed command. + +Enabling the importer +--------------------- + +The YAML importer is opt-in. Register it in +:ref:`config:dag_processor__dag_importer_configs`: + +.. code-block:: ini + + [dag_processor] + dag_importer_configs = [{"classpath": "airflow.sdk.importers.yaml_importer.YamlDagImporter"}] + +The importer handles ``.yaml`` and ``.yml`` files that declare a Dag ``$schema`` +(see below). For per-bundle registration and other options, see +:doc:`../administration-and-deployment/dag-importers`. + +The shape of a document +----------------------- + +``$schema`` (required) + The format version the document is written against, given as the URL of a + published schema. The version it names selects the rules the document is + parsed with. ``$schema`` is also how Airflow recognizes the file + as a Dag; a file without it is silently skipped. Editors that understand + ``$schema`` will offer validation and completion with no extra setup. + +Dag-level keys + Attribute of the Dag sit at the top level, such as ``dag_id``, ``schedule``, + ``params``, etc. An unknown key is reported as an error. + +``tasks`` + A list of tasks. Each task has an ``id`` and exactly one body. + +Task body: ``uses`` or ``run`` + ``uses:`` names a ready-made operator by its fully-qualified Python import + path (e.g. ``airflow.providers.standard.operators.bash.BashOperator``). + ``run:`` holds the arguments of a code task you implement elsewhere (see + `Code tasks`_ below). A task may have one or the other. + +``with`` + Operator-specific arguments. These are what you would pass to the operator's + constructor. Read on to understand the value grammar you can use in these + arguments. + +Task-level keys + Scheduling and common attributes sit next to the body, distinct from + ``with:``. These include things like ``retries``, ``trigger_rule``, + ``queue``, etc. Any key that is not one of the structural keys (``id``, + ``uses``, ``run``, ``with``, ``needs``, ``extends``) is passed to the + operator type, which validates it. + +``needs`` + A list of upstream tasks (referenced by ``id``) that must run before this + one. This describes an explicit dependency edge; XCom references in + ``with:`` or ``run:`` are discovered automatically and do not need to be + repeated here. + + +Templates and extends +--------------------- + +Tasks often share arguments. You can define a named partial task under the +top-level ``templates:`` key, and pull it into a task with ``extends``: + +.. code-block:: yaml + + templates: + retryable: + retries: 2 + shell: + extends: [retryable] # a template can extend other templates + uses: airflow.providers.standard.operators.bash.BashOperator + with: + cwd: /opt/airflow/scripts + + tasks: + - id: extract + extends: [shell] # pull the template into this task + with: + bash_command: "extract.sh" + - id: load + extends: [shell] + with: + bash_command: "load.sh" + +Both tasks run a ``BashOperator`` with ``retries: 2`` and the shared ``cwd``, +and add only their own ``bash_command``. A template may set anything a task can, +and may itself ``extends`` other templates. + +The templates named in ``extends`` are applied left to right, and the task's own +inline fields are applied last. Merging is shallow: ``with:`` and ``run:`` are +combined one key at a time, while every other field is taken as a whole rather +than deep-merged. + +The assembled task is validated against the real operator, so a field the +operator does not accept is reported as an error. It is also an error if a +non-existent template is referenced. + + +Values are literal by default +----------------------------- + +A value under ``with:`` (or ``run:``) is taken literally, unless it is a +single-key object whose key is a reserved ``$``-marker. For example, +``"echo foo.sh"`` and ``"{{ not rendered }}"`` specifies exactly that string, +not a template, even if ``bash_command`` is listed on the operator as a template +field. + +Four markers opt a value into special handling: + +``{$t: "..."}`` is a template string + A Jinja template, rendered when the task runs. This is the only way a value + is templated: ``{$t: "{{ ds }}"}``, ``{$t: "{{ params.region }}"}``, + ``{$t: "{{ conn.my_db.host }}"}``. The long form ``{$template: "..."}`` is + also accepted. + +``{$x: id}`` references an upstream output + The XCom output of an upstream task. ``{$x: extract}`` references + ``extract``'s return value, and ``{$x: {task: extract, key: rows}}`` a + specific key. Binding a value this way also creates the dependency edge + implicitly, so a task that consumes another's output does not need to also + list it in ``needs:``. The long form is ``{$xcom: ...}``. + +``{$f: path}`` reads a file's contents + The contents of a file stored next to the document is read and used. + ``{$t: {$f: path}}`` allows reading the file and then rendering its contents + as a template. The long form is ``{$file: path}``. A file is loaded only + when you ask with ``$f``; it is never inferred from a filename suffix. + +``{$const: value}`` is a forced literal + An escape for the rare case that real data is shaped exactly like a marker. + ``{$const: {$x: 1}}`` is the literal dict ``{"$x": 1}``, not an XCom. + +Markers nest. A marker inside a list, or inside a nested mapping such as an +``env`` dict, is resolved by the same rules, so references and templates compose +through structure: + +.. code-block:: yaml + + with: + env: + RUN_DATE: + $t: "{{ ds }}" # rendered + REGION: us-east-1 # literal + +Dates and datetimes +------------------- + +A date or datetime argument accepts an ISO-8601 string or a YAML-native date, +and they mean the same thing: + +.. code-block:: yaml + + start_date: "2024-01-01" # ISO-8601 string + start_date: 2024-01-01 # YAML date + end_date: "2024-06-01T12:00:00" # ISO-8601 datetime string + +These are converted to a Python ``datetime`` before the Dag or task is built. A bare date becomes +midnight, and an ISO-8601 string is parsed. + +The conversion is applied only to an argument that expects a datetime. This is +decided from the operator's or Dag's argument **type annotation**. + +Using your own operators +------------------------ + +``uses:`` takes any operator's fully-qualified Python import path, so your own +operator class can work exactly like a provider's. + +.. code-block:: yaml + + - id: ingest + uses: my_company.operators.IngestOperator + trigger_rule: all_done # base arguments at the task-level + with: # your operator's own arguments + source: s3://bucket/incoming + cutoff: 2024-01-01 + +An operator takes two kinds of argument, kept in separate places. Your operator's own constructor +arguments go under ``with:``. The arguments every operator inherits from the base operator -- +``retries``, ``retry_delay``, ``trigger_rule``, ``queue``, ``pool``, ``priority_weight``, +``depends_on_past``, ``start_date``, and the rest of the task model -- go at the task level, beside +the body. Placing a base argument under ``with:``, or an operator-specific one at the task level, is +an error, so each argument has a single home. + +The task is constructed roughly like this: + +.. code-block:: python + + IngestOperator( + task_id="ingest", + source="s3://bucket/incoming", + cutoff=datetime.datetime(2024, 1, 1, 0, 0), + trigger_rule="all_done", + ) + +Templated argument rendering +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +A task argument is literal by default. For one to be rendered as a Jinja +template, two things must satisfy: the operator must allow that field to be +templated (for Python operators, by specifying it in ``template_fields``), and +the Dag author must use the ``{$t: ...}`` syntax. A value not marked explicitly +stays literal. Try to use the template syntax on a field not specified by the +operator results in an error. + +A field leave unset keeps the operator's own default, including any templating +the operator applies to that default. A value is also never read from a file +because of its name or suffix (Python's ``template_ext`` does not apply); a file +is loaded only where you ask with ``{$f: ...}``. + +Literal values inside a rendered container +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +A literal also stays literal when it sits in the same list or mapping as a +``{$t: ...}`` or ``{$x: ...}``. The marked entry renders, and the plain entries +beside it do not. Most operators receive the finished value and never notice. An +operator whose ``__init__`` reads the individual entries of such a container may +notice edge cases, but operator authors should be aware that argument processing +in ``__init__`` is generally considered bad practice in the first place. Such +processing should generally be done in ``execute`` instead, which is always +called after template rendering has be performed. + +Accepting a date argument +~~~~~~~~~~~~~~~~~~~~~~~~~ + +Whether a value is converted to a ``datetime`` depends on how the receiving +argument is typed. An argument whose type annotation names a datetime type +receives the conversion described in `Dates and datetimes`_; every other +argument is left as written. To let your operator take a YAML date, annotate the +argument: + +.. code-block:: python + + import datetime + + from airflow.sdk import BaseOperator + + + class IngestOperator(BaseOperator): + def __init__(self, *, cutoff: datetime.datetime | None = None, **kwargs): + super().__init__(**kwargs) + self.cutoff = cutoff + +With the annotation above, ``cutoff: 2024-01-01`` from YAML arrives as +``datetime.datetime(2024, 1, 1, 0, 0)``. The annotation is matched by name, so +``datetime``, ``datetime.datetime``, ``pendulum.DateTime``, and a union such as +``datetime | None`` are all recognized; a datetime type imported under another +alias is not. + +Reserved arguments +~~~~~~~~~~~~~~~~~~ + +A few names are reserved and cannot be set on a task. The task id comes from +``id``, not ``task_id``; arguments ``dag``, ``task_group``, and ``default_args`` +are not accepted. Use ``templates:`` and ``extends:`` for shared defaults in +place of ``default_args``. (Support for task groups is planned for a future +release.) + +Scheduling +---------- + +``schedule:`` accepts the forms Airflow already understands, chosen by shape: + +* a cron string (``"0 3 * * *"``) or a preset (``"@daily"``); +* ``null`` or an omitted key, for a Dag that is only triggered manually or + externally; +* an object ``{uses: <timetable import path>, with: {...}}`` to construct any + timetable, the same ``uses`` / ``with`` idea used for operators. Review Comment: `with:` here doesn't go through the task value grammar or the date coercion: `_build_schedule` passes it straight to the timetable constructor. So `DeltaTriggerTimetable` (its `delta` must be a timedelta) can't be built from YAML today. Maybe narrow "any timetable", or note that markers and date conversion don't apply under a schedule's `with:`. ########## airflow-core/docs/authoring-and-scheduling/yaml-dags.rst: ########## @@ -0,0 +1,349 @@ + .. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +.. _yaml-dags: + +Writing Dags in YAML +==================== + +|experimental| + +.. versionadded:: 3.4.0 + +Airflow can build a Dag from a YAML document. One YAML document describes one +Dag, and the YAML Dag importer turns it into the same Dag structure that Airflow +runs like any other source. The tasks can be ordinary Airflow operators, so +every provider operator is available without writing code. + +A document is declarative. Everything in it is a value, and the shape of each +value decides what it means. Since the format round-trips through JSON, a +document may equally be written as JSON. + +The following is a example YAML file defining a Dag: + +.. code-block:: yaml + + $schema: https://airflow.apache.org/schemas/dag/2026-10-30.json + dag_id: daily_report + schedule: "0 3 * * *" + start_date: 2024-01-01 + + tasks: + - id: extract + uses: airflow.providers.standard.operators.bash.BashOperator + with: + bash_command: + $t: "extract.sh {{ ds }}" + + - id: publish + uses: airflow.providers.standard.operators.bash.BashOperator + needs: [extract] + with: + bash_command: "publish.sh" + +This Dag runs every day at 03:00, The first task ``extract`` renders the date +into its command. The second ``publish`` waits for ``extract`` through +``needs`` and runs a fixed command. + +Enabling the importer +--------------------- + +The YAML importer is opt-in. Register it in +:ref:`config:dag_processor__dag_importer_configs`: + +.. code-block:: ini + + [dag_processor] + dag_importer_configs = [{"classpath": "airflow.sdk.importers.yaml_importer.YamlDagImporter"}] + +The importer handles ``.yaml`` and ``.yml`` files that declare a Dag ``$schema`` +(see below). For per-bundle registration and other options, see +:doc:`../administration-and-deployment/dag-importers`. + +The shape of a document +----------------------- + +``$schema`` (required) + The format version the document is written against, given as the URL of a + published schema. The version it names selects the rules the document is + parsed with. ``$schema`` is also how Airflow recognizes the file + as a Dag; a file without it is silently skipped. Editors that understand + ``$schema`` will offer validation and completion with no extra setup. + +Dag-level keys + Attribute of the Dag sit at the top level, such as ``dag_id``, ``schedule``, + ``params``, etc. An unknown key is reported as an error. + +``tasks`` + A list of tasks. Each task has an ``id`` and exactly one body. + +Task body: ``uses`` or ``run`` + ``uses:`` names a ready-made operator by its fully-qualified Python import + path (e.g. ``airflow.providers.standard.operators.bash.BashOperator``). + ``run:`` holds the arguments of a code task you implement elsewhere (see + `Code tasks`_ below). A task may have one or the other. + +``with`` + Operator-specific arguments. These are what you would pass to the operator's + constructor. Read on to understand the value grammar you can use in these + arguments. + +Task-level keys + Scheduling and common attributes sit next to the body, distinct from + ``with:``. These include things like ``retries``, ``trigger_rule``, + ``queue``, etc. Any key that is not one of the structural keys (``id``, + ``uses``, ``run``, ``with``, ``needs``, ``extends``) is passed to the + operator type, which validates it. + +``needs`` + A list of upstream tasks (referenced by ``id``) that must run before this + one. This describes an explicit dependency edge; XCom references in + ``with:`` or ``run:`` are discovered automatically and do not need to be + repeated here. + + +Templates and extends +--------------------- + +Tasks often share arguments. You can define a named partial task under the +top-level ``templates:`` key, and pull it into a task with ``extends``: + +.. code-block:: yaml + + templates: + retryable: + retries: 2 + shell: + extends: [retryable] # a template can extend other templates + uses: airflow.providers.standard.operators.bash.BashOperator + with: + cwd: /opt/airflow/scripts + + tasks: + - id: extract + extends: [shell] # pull the template into this task + with: + bash_command: "extract.sh" + - id: load + extends: [shell] + with: + bash_command: "load.sh" + +Both tasks run a ``BashOperator`` with ``retries: 2`` and the shared ``cwd``, +and add only their own ``bash_command``. A template may set anything a task can, +and may itself ``extends`` other templates. + +The templates named in ``extends`` are applied left to right, and the task's own +inline fields are applied last. Merging is shallow: ``with:`` and ``run:`` are +combined one key at a time, while every other field is taken as a whole rather +than deep-merged. + +The assembled task is validated against the real operator, so a field the +operator does not accept is reported as an error. It is also an error if a +non-existent template is referenced. + + +Values are literal by default +----------------------------- + +A value under ``with:`` (or ``run:``) is taken literally, unless it is a +single-key object whose key is a reserved ``$``-marker. For example, +``"echo foo.sh"`` and ``"{{ not rendered }}"`` specifies exactly that string, +not a template, even if ``bash_command`` is listed on the operator as a template +field. + +Four markers opt a value into special handling: + +``{$t: "..."}`` is a template string + A Jinja template, rendered when the task runs. This is the only way a value + is templated: ``{$t: "{{ ds }}"}``, ``{$t: "{{ params.region }}"}``, + ``{$t: "{{ conn.my_db.host }}"}``. The long form ``{$template: "..."}`` is + also accepted. + +``{$x: id}`` references an upstream output + The XCom output of an upstream task. ``{$x: extract}`` references + ``extract``'s return value, and ``{$x: {task: extract, key: rows}}`` a + specific key. Binding a value this way also creates the dependency edge + implicitly, so a task that consumes another's output does not need to also + list it in ``needs:``. The long form is ``{$xcom: ...}``. + +``{$f: path}`` reads a file's contents + The contents of a file stored next to the document is read and used. + ``{$t: {$f: path}}`` allows reading the file and then rendering its contents + as a template. The long form is ``{$file: path}``. A file is loaded only + when you ask with ``$f``; it is never inferred from a filename suffix. + +``{$const: value}`` is a forced literal + An escape for the rare case that real data is shaped exactly like a marker. + ``{$const: {$x: 1}}`` is the literal dict ``{"$x": 1}``, not an XCom. + +Markers nest. A marker inside a list, or inside a nested mapping such as an +``env`` dict, is resolved by the same rules, so references and templates compose +through structure: + +.. code-block:: yaml + + with: + env: + RUN_DATE: + $t: "{{ ds }}" # rendered + REGION: us-east-1 # literal + +Dates and datetimes +------------------- + +A date or datetime argument accepts an ISO-8601 string or a YAML-native date, +and they mean the same thing: + +.. code-block:: yaml + + start_date: "2024-01-01" # ISO-8601 string + start_date: 2024-01-01 # YAML date + end_date: "2024-06-01T12:00:00" # ISO-8601 datetime string + +These are converted to a Python ``datetime`` before the Dag or task is built. A bare date becomes +midnight, and an ISO-8601 string is parsed. + +The conversion is applied only to an argument that expects a datetime. This is +decided from the operator's or Dag's argument **type annotation**. + +Using your own operators +------------------------ + +``uses:`` takes any operator's fully-qualified Python import path, so your own +operator class can work exactly like a provider's. + +.. code-block:: yaml + + - id: ingest + uses: my_company.operators.IngestOperator + trigger_rule: all_done # base arguments at the task-level + with: # your operator's own arguments + source: s3://bucket/incoming + cutoff: 2024-01-01 + +An operator takes two kinds of argument, kept in separate places. Your operator's own constructor +arguments go under ``with:``. The arguments every operator inherits from the base operator -- +``retries``, ``retry_delay``, ``trigger_rule``, ``queue``, ``pool``, ``priority_weight``, +``depends_on_past``, ``start_date``, and the rest of the task model -- go at the task level, beside +the body. Placing a base argument under ``with:``, or an operator-specific one at the task level, is +an error, so each argument has a single home. + +The task is constructed roughly like this: + +.. code-block:: python + + IngestOperator( + task_id="ingest", + source="s3://bucket/incoming", + cutoff=datetime.datetime(2024, 1, 1, 0, 0), + trigger_rule="all_done", + ) + +Templated argument rendering +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +A task argument is literal by default. For one to be rendered as a Jinja +template, two things must satisfy: the operator must allow that field to be +templated (for Python operators, by specifying it in ``template_fields``), and +the Dag author must use the ``{$t: ...}`` syntax. A value not marked explicitly +stays literal. Try to use the template syntax on a field not specified by the +operator results in an error. Review Comment: This applies to `$x` and `$f` too, not only `$t`. `_field_markers` flags all three, and `_build_operator` raises "value grammar markers are only allowed on templated fields" for any of them; only `$const` is allowed elsewhere. Worth saying once in the markers section. It also affects the custom operator example above: `IngestOperator` declares no `template_fields`, so `source: {$x: extract}` on it would fail to import. ########## airflow-core/docs/authoring-and-scheduling/yaml-dags.rst: ########## @@ -0,0 +1,349 @@ + .. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +.. _yaml-dags: + +Writing Dags in YAML +==================== + +|experimental| + +.. versionadded:: 3.4.0 + +Airflow can build a Dag from a YAML document. One YAML document describes one +Dag, and the YAML Dag importer turns it into the same Dag structure that Airflow +runs like any other source. The tasks can be ordinary Airflow operators, so +every provider operator is available without writing code. + +A document is declarative. Everything in it is a value, and the shape of each +value decides what it means. Since the format round-trips through JSON, a +document may equally be written as JSON. + +The following is a example YAML file defining a Dag: + +.. code-block:: yaml + + $schema: https://airflow.apache.org/schemas/dag/2026-10-30.json + dag_id: daily_report + schedule: "0 3 * * *" + start_date: 2024-01-01 + + tasks: + - id: extract + uses: airflow.providers.standard.operators.bash.BashOperator + with: + bash_command: + $t: "extract.sh {{ ds }}" + + - id: publish + uses: airflow.providers.standard.operators.bash.BashOperator + needs: [extract] + with: + bash_command: "publish.sh" + +This Dag runs every day at 03:00, The first task ``extract`` renders the date +into its command. The second ``publish`` waits for ``extract`` through +``needs`` and runs a fixed command. + +Enabling the importer +--------------------- + +The YAML importer is opt-in. Register it in +:ref:`config:dag_processor__dag_importer_configs`: + +.. code-block:: ini + + [dag_processor] + dag_importer_configs = [{"classpath": "airflow.sdk.importers.yaml_importer.YamlDagImporter"}] + +The importer handles ``.yaml`` and ``.yml`` files that declare a Dag ``$schema`` +(see below). For per-bundle registration and other options, see +:doc:`../administration-and-deployment/dag-importers`. + +The shape of a document +----------------------- + +``$schema`` (required) + The format version the document is written against, given as the URL of a + published schema. The version it names selects the rules the document is + parsed with. ``$schema`` is also how Airflow recognizes the file + as a Dag; a file without it is silently skipped. Editors that understand + ``$schema`` will offer validation and completion with no extra setup. + +Dag-level keys + Attribute of the Dag sit at the top level, such as ``dag_id``, ``schedule``, + ``params``, etc. An unknown key is reported as an error. + +``tasks`` + A list of tasks. Each task has an ``id`` and exactly one body. + +Task body: ``uses`` or ``run`` + ``uses:`` names a ready-made operator by its fully-qualified Python import + path (e.g. ``airflow.providers.standard.operators.bash.BashOperator``). + ``run:`` holds the arguments of a code task you implement elsewhere (see + `Code tasks`_ below). A task may have one or the other. + +``with`` + Operator-specific arguments. These are what you would pass to the operator's + constructor. Read on to understand the value grammar you can use in these + arguments. + +Task-level keys + Scheduling and common attributes sit next to the body, distinct from + ``with:``. These include things like ``retries``, ``trigger_rule``, + ``queue``, etc. Any key that is not one of the structural keys (``id``, + ``uses``, ``run``, ``with``, ``needs``, ``extends``) is passed to the + operator type, which validates it. + +``needs`` + A list of upstream tasks (referenced by ``id``) that must run before this + one. This describes an explicit dependency edge; XCom references in + ``with:`` or ``run:`` are discovered automatically and do not need to be + repeated here. + + +Templates and extends +--------------------- + +Tasks often share arguments. You can define a named partial task under the +top-level ``templates:`` key, and pull it into a task with ``extends``: + +.. code-block:: yaml + + templates: + retryable: + retries: 2 + shell: + extends: [retryable] # a template can extend other templates + uses: airflow.providers.standard.operators.bash.BashOperator + with: + cwd: /opt/airflow/scripts + + tasks: + - id: extract + extends: [shell] # pull the template into this task + with: + bash_command: "extract.sh" + - id: load + extends: [shell] + with: + bash_command: "load.sh" + +Both tasks run a ``BashOperator`` with ``retries: 2`` and the shared ``cwd``, +and add only their own ``bash_command``. A template may set anything a task can, +and may itself ``extends`` other templates. + +The templates named in ``extends`` are applied left to right, and the task's own +inline fields are applied last. Merging is shallow: ``with:`` and ``run:`` are +combined one key at a time, while every other field is taken as a whole rather +than deep-merged. + +The assembled task is validated against the real operator, so a field the +operator does not accept is reported as an error. It is also an error if a +non-existent template is referenced. + + +Values are literal by default +----------------------------- + +A value under ``with:`` (or ``run:``) is taken literally, unless it is a +single-key object whose key is a reserved ``$``-marker. For example, +``"echo foo.sh"`` and ``"{{ not rendered }}"`` specifies exactly that string, +not a template, even if ``bash_command`` is listed on the operator as a template +field. + +Four markers opt a value into special handling: + +``{$t: "..."}`` is a template string + A Jinja template, rendered when the task runs. This is the only way a value + is templated: ``{$t: "{{ ds }}"}``, ``{$t: "{{ params.region }}"}``, + ``{$t: "{{ conn.my_db.host }}"}``. The long form ``{$template: "..."}`` is + also accepted. + +``{$x: id}`` references an upstream output + The XCom output of an upstream task. ``{$x: extract}`` references + ``extract``'s return value, and ``{$x: {task: extract, key: rows}}`` a + specific key. Binding a value this way also creates the dependency edge + implicitly, so a task that consumes another's output does not need to also + list it in ``needs:``. The long form is ``{$xcom: ...}``. + +``{$f: path}`` reads a file's contents + The contents of a file stored next to the document is read and used. + ``{$t: {$f: path}}`` allows reading the file and then rendering its contents + as a template. The long form is ``{$file: path}``. A file is loaded only + when you ask with ``$f``; it is never inferred from a filename suffix. + +``{$const: value}`` is a forced literal + An escape for the rare case that real data is shaped exactly like a marker. + ``{$const: {$x: 1}}`` is the literal dict ``{"$x": 1}``, not an XCom. + +Markers nest. A marker inside a list, or inside a nested mapping such as an +``env`` dict, is resolved by the same rules, so references and templates compose +through structure: + +.. code-block:: yaml + + with: + env: + RUN_DATE: + $t: "{{ ds }}" # rendered + REGION: us-east-1 # literal + +Dates and datetimes +------------------- + +A date or datetime argument accepts an ISO-8601 string or a YAML-native date, +and they mean the same thing: + +.. code-block:: yaml + + start_date: "2024-01-01" # ISO-8601 string + start_date: 2024-01-01 # YAML date + end_date: "2024-06-01T12:00:00" # ISO-8601 datetime string + +These are converted to a Python ``datetime`` before the Dag or task is built. A bare date becomes +midnight, and an ISO-8601 string is parsed. + +The conversion is applied only to an argument that expects a datetime. This is +decided from the operator's or Dag's argument **type annotation**. + +Using your own operators +------------------------ + +``uses:`` takes any operator's fully-qualified Python import path, so your own +operator class can work exactly like a provider's. + +.. code-block:: yaml + + - id: ingest + uses: my_company.operators.IngestOperator + trigger_rule: all_done # base arguments at the task-level + with: # your operator's own arguments + source: s3://bucket/incoming + cutoff: 2024-01-01 + +An operator takes two kinds of argument, kept in separate places. Your operator's own constructor +arguments go under ``with:``. The arguments every operator inherits from the base operator -- +``retries``, ``retry_delay``, ``trigger_rule``, ``queue``, ``pool``, ``priority_weight``, +``depends_on_past``, ``start_date``, and the rest of the task model -- go at the task level, beside +the body. Placing a base argument under ``with:``, or an operator-specific one at the task level, is +an error, so each argument has a single home. + +The task is constructed roughly like this: + +.. code-block:: python + + IngestOperator( + task_id="ingest", + source="s3://bucket/incoming", + cutoff=datetime.datetime(2024, 1, 1, 0, 0), + trigger_rule="all_done", + ) + +Templated argument rendering +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +A task argument is literal by default. For one to be rendered as a Jinja +template, two things must satisfy: the operator must allow that field to be +templated (for Python operators, by specifying it in ``template_fields``), and +the Dag author must use the ``{$t: ...}`` syntax. A value not marked explicitly +stays literal. Try to use the template syntax on a field not specified by the +operator results in an error. + +A field leave unset keeps the operator's own default, including any templating +the operator applies to that default. A value is also never read from a file +because of its name or suffix (Python's ``template_ext`` does not apply); a file +is loaded only where you ask with ``{$f: ...}``. + +Literal values inside a rendered container +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +A literal also stays literal when it sits in the same list or mapping as a +``{$t: ...}`` or ``{$x: ...}``. The marked entry renders, and the plain entries +beside it do not. Most operators receive the finished value and never notice. An +operator whose ``__init__`` reads the individual entries of such a container may +notice edge cases, but operator authors should be aware that argument processing +in ``__init__`` is generally considered bad practice in the first place. Such +processing should generally be done in ``execute`` instead, which is always +called after template rendering has be performed. + +Accepting a date argument +~~~~~~~~~~~~~~~~~~~~~~~~~ + +Whether a value is converted to a ``datetime`` depends on how the receiving +argument is typed. An argument whose type annotation names a datetime type +receives the conversion described in `Dates and datetimes`_; every other +argument is left as written. To let your operator take a YAML date, annotate the +argument: + +.. code-block:: python + + import datetime + + from airflow.sdk import BaseOperator + + + class IngestOperator(BaseOperator): + def __init__(self, *, cutoff: datetime.datetime | None = None, **kwargs): + super().__init__(**kwargs) + self.cutoff = cutoff + +With the annotation above, ``cutoff: 2024-01-01`` from YAML arrives as +``datetime.datetime(2024, 1, 1, 0, 0)``. The annotation is matched by name, so +``datetime``, ``datetime.datetime``, ``pendulum.DateTime``, and a union such as +``datetime | None`` are all recognized; a datetime type imported under another +alias is not. + +Reserved arguments +~~~~~~~~~~~~~~~~~~ + +A few names are reserved and cannot be set on a task. The task id comes from +``id``, not ``task_id``; arguments ``dag``, ``task_group``, and ``default_args`` +are not accepted. Use ``templates:`` and ``extends:`` for shared defaults in +place of ``default_args``. (Support for task groups is planned for a future +release.) + +Scheduling +---------- + +``schedule:`` accepts the forms Airflow already understands, chosen by shape: + +* a cron string (``"0 3 * * *"``) or a preset (``"@daily"``); +* ``null`` or an omitted key, for a Dag that is only triggered manually or + externally; +* an object ``{uses: <timetable import path>, with: {...}}`` to construct any + timetable, the same ``uses`` / ``with`` idea used for operators. + +Support for asset-triggering is planned for a future release. + +Code tasks +---------- + +A task can run your own code instead of a ready-made operator with ``run:``, +which holds the code's arguments directly: + +.. code-block:: yaml + + - id: transform + run: + source: {$x: extract} + factor: 2 + +A ``run:`` task is built into a stub that a Task SDK runtime executes. See +:doc:`language-sdks/index` to understand how to write a task handler for runtime +execution. A top-level ``{$x: ...}`` argument binds an upstream task's output; Review Comment: Two things on this section. The example can't run as written: `run:` builds a `@task.stub`, whose `execute` raises if it runs in Python, and nothing routes it to a coordinator without a task-level `queue:` mapped in `[sdk] queue_to_coordinator`. Adding a `queue:` to the example would help. Also, line 163 says markers work under `run:`, but `_reject_unbindable_run_marker` rejects `$t` anywhere, a keyed `$x`, and a nested `$x`. Could this sentence say only an unkeyed `{$x: task}` given as its own argument binds, and `$f`/`$const` resolve to literals? ########## airflow-core/docs/authoring-and-scheduling/yaml-dags.rst: ########## @@ -0,0 +1,349 @@ + .. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +.. _yaml-dags: + +Writing Dags in YAML +==================== + +|experimental| + +.. versionadded:: 3.4.0 + +Airflow can build a Dag from a YAML document. One YAML document describes one +Dag, and the YAML Dag importer turns it into the same Dag structure that Airflow +runs like any other source. The tasks can be ordinary Airflow operators, so +every provider operator is available without writing code. + +A document is declarative. Everything in it is a value, and the shape of each +value decides what it means. Since the format round-trips through JSON, a +document may equally be written as JSON. + +The following is a example YAML file defining a Dag: + +.. code-block:: yaml + + $schema: https://airflow.apache.org/schemas/dag/2026-10-30.json + dag_id: daily_report + schedule: "0 3 * * *" + start_date: 2024-01-01 + + tasks: + - id: extract + uses: airflow.providers.standard.operators.bash.BashOperator + with: + bash_command: + $t: "extract.sh {{ ds }}" + + - id: publish + uses: airflow.providers.standard.operators.bash.BashOperator + needs: [extract] + with: + bash_command: "publish.sh" + +This Dag runs every day at 03:00, The first task ``extract`` renders the date +into its command. The second ``publish`` waits for ``extract`` through +``needs`` and runs a fixed command. + +Enabling the importer +--------------------- + +The YAML importer is opt-in. Register it in +:ref:`config:dag_processor__dag_importer_configs`: + +.. code-block:: ini + + [dag_processor] + dag_importer_configs = [{"classpath": "airflow.sdk.importers.yaml_importer.YamlDagImporter"}] + +The importer handles ``.yaml`` and ``.yml`` files that declare a Dag ``$schema`` +(see below). For per-bundle registration and other options, see +:doc:`../administration-and-deployment/dag-importers`. + +The shape of a document +----------------------- + +``$schema`` (required) + The format version the document is written against, given as the URL of a + published schema. The version it names selects the rules the document is + parsed with. ``$schema`` is also how Airflow recognizes the file + as a Dag; a file without it is silently skipped. Editors that understand + ``$schema`` will offer validation and completion with no extra setup. Review Comment: `https://airflow.apache.org/schemas/dag/2026-10-30.json` returns 404 right now, so editors have nothing to fetch, and "the URL of a published schema" on line 81 isn't true yet either. Is publishing it to airflow-site planned for the release? If not, I'd drop the editor sentence until it is. The existing `schemas/dag-serialization/v2.json` link in dag-serialization.rst also 404s, so I don't think there's a publishing step to rely on. ########## airflow-core/docs/authoring-and-scheduling/yaml-dags.rst: ########## @@ -0,0 +1,349 @@ + .. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +.. _yaml-dags: + +Writing Dags in YAML +==================== + +|experimental| + +.. versionadded:: 3.4.0 + +Airflow can build a Dag from a YAML document. One YAML document describes one +Dag, and the YAML Dag importer turns it into the same Dag structure that Airflow +runs like any other source. The tasks can be ordinary Airflow operators, so +every provider operator is available without writing code. + +A document is declarative. Everything in it is a value, and the shape of each +value decides what it means. Since the format round-trips through JSON, a +document may equally be written as JSON. + +The following is a example YAML file defining a Dag: + +.. code-block:: yaml + + $schema: https://airflow.apache.org/schemas/dag/2026-10-30.json + dag_id: daily_report + schedule: "0 3 * * *" + start_date: 2024-01-01 + + tasks: + - id: extract + uses: airflow.providers.standard.operators.bash.BashOperator + with: + bash_command: + $t: "extract.sh {{ ds }}" + + - id: publish + uses: airflow.providers.standard.operators.bash.BashOperator + needs: [extract] + with: + bash_command: "publish.sh" + +This Dag runs every day at 03:00, The first task ``extract`` renders the date +into its command. The second ``publish`` waits for ``extract`` through +``needs`` and runs a fixed command. + +Enabling the importer +--------------------- + +The YAML importer is opt-in. Register it in +:ref:`config:dag_processor__dag_importer_configs`: + +.. code-block:: ini + + [dag_processor] + dag_importer_configs = [{"classpath": "airflow.sdk.importers.yaml_importer.YamlDagImporter"}] + +The importer handles ``.yaml`` and ``.yml`` files that declare a Dag ``$schema`` +(see below). For per-bundle registration and other options, see +:doc:`../administration-and-deployment/dag-importers`. + +The shape of a document +----------------------- + +``$schema`` (required) + The format version the document is written against, given as the URL of a + published schema. The version it names selects the rules the document is + parsed with. ``$schema`` is also how Airflow recognizes the file + as a Dag; a file without it is silently skipped. Editors that understand + ``$schema`` will offer validation and completion with no extra setup. + +Dag-level keys + Attribute of the Dag sit at the top level, such as ``dag_id``, ``schedule``, + ``params``, etc. An unknown key is reported as an error. + +``tasks`` + A list of tasks. Each task has an ``id`` and exactly one body. + +Task body: ``uses`` or ``run`` + ``uses:`` names a ready-made operator by its fully-qualified Python import + path (e.g. ``airflow.providers.standard.operators.bash.BashOperator``). + ``run:`` holds the arguments of a code task you implement elsewhere (see + `Code tasks`_ below). A task may have one or the other. + +``with`` + Operator-specific arguments. These are what you would pass to the operator's + constructor. Read on to understand the value grammar you can use in these + arguments. + +Task-level keys + Scheduling and common attributes sit next to the body, distinct from + ``with:``. These include things like ``retries``, ``trigger_rule``, + ``queue``, etc. Any key that is not one of the structural keys (``id``, + ``uses``, ``run``, ``with``, ``needs``, ``extends``) is passed to the + operator type, which validates it. + +``needs`` + A list of upstream tasks (referenced by ``id``) that must run before this + one. This describes an explicit dependency edge; XCom references in + ``with:`` or ``run:`` are discovered automatically and do not need to be + repeated here. + + +Templates and extends +--------------------- + +Tasks often share arguments. You can define a named partial task under the +top-level ``templates:`` key, and pull it into a task with ``extends``: + +.. code-block:: yaml + + templates: + retryable: + retries: 2 + shell: + extends: [retryable] # a template can extend other templates + uses: airflow.providers.standard.operators.bash.BashOperator + with: + cwd: /opt/airflow/scripts + + tasks: + - id: extract + extends: [shell] # pull the template into this task + with: + bash_command: "extract.sh" + - id: load + extends: [shell] + with: + bash_command: "load.sh" + +Both tasks run a ``BashOperator`` with ``retries: 2`` and the shared ``cwd``, +and add only their own ``bash_command``. A template may set anything a task can, +and may itself ``extends`` other templates. + +The templates named in ``extends`` are applied left to right, and the task's own +inline fields are applied last. Merging is shallow: ``with:`` and ``run:`` are +combined one key at a time, while every other field is taken as a whole rather +than deep-merged. + +The assembled task is validated against the real operator, so a field the +operator does not accept is reported as an error. It is also an error if a +non-existent template is referenced. + + +Values are literal by default +----------------------------- + +A value under ``with:`` (or ``run:``) is taken literally, unless it is a +single-key object whose key is a reserved ``$``-marker. For example, Review Comment: The check is wider than a single-key object whose key is a marker. `Literal._reject_reserved_keys` rejects any mapping with a `$`-prefixed key, so `env: {"$HOME": x, Y: z}` or a JSON Schema carrying `$ref` fails to parse. That changes the `$const` description on line 191 too, since it's needed for any `$` key, not only data shaped exactly like a marker. ########## airflow-core/docs/authoring-and-scheduling/yaml-dags.rst: ########## @@ -0,0 +1,349 @@ + .. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +.. _yaml-dags: + +Writing Dags in YAML +==================== + +|experimental| + +.. versionadded:: 3.4.0 + +Airflow can build a Dag from a YAML document. One YAML document describes one +Dag, and the YAML Dag importer turns it into the same Dag structure that Airflow +runs like any other source. The tasks can be ordinary Airflow operators, so +every provider operator is available without writing code. + +A document is declarative. Everything in it is a value, and the shape of each +value decides what it means. Since the format round-trips through JSON, a +document may equally be written as JSON. + +The following is a example YAML file defining a Dag: + +.. code-block:: yaml + + $schema: https://airflow.apache.org/schemas/dag/2026-10-30.json + dag_id: daily_report + schedule: "0 3 * * *" + start_date: 2024-01-01 + + tasks: + - id: extract + uses: airflow.providers.standard.operators.bash.BashOperator + with: + bash_command: + $t: "extract.sh {{ ds }}" + + - id: publish + uses: airflow.providers.standard.operators.bash.BashOperator + needs: [extract] + with: + bash_command: "publish.sh" + +This Dag runs every day at 03:00, The first task ``extract`` renders the date +into its command. The second ``publish`` waits for ``extract`` through +``needs`` and runs a fixed command. + +Enabling the importer +--------------------- + +The YAML importer is opt-in. Register it in +:ref:`config:dag_processor__dag_importer_configs`: + +.. code-block:: ini + + [dag_processor] + dag_importer_configs = [{"classpath": "airflow.sdk.importers.yaml_importer.YamlDagImporter"}] + +The importer handles ``.yaml`` and ``.yml`` files that declare a Dag ``$schema`` +(see below). For per-bundle registration and other options, see +:doc:`../administration-and-deployment/dag-importers`. + +The shape of a document +----------------------- + +``$schema`` (required) + The format version the document is written against, given as the URL of a + published schema. The version it names selects the rules the document is + parsed with. ``$schema`` is also how Airflow recognizes the file + as a Dag; a file without it is silently skipped. Editors that understand + ``$schema`` will offer validation and completion with no extra setup. + +Dag-level keys + Attribute of the Dag sit at the top level, such as ``dag_id``, ``schedule``, + ``params``, etc. An unknown key is reported as an error. + +``tasks`` + A list of tasks. Each task has an ``id`` and exactly one body. + +Task body: ``uses`` or ``run`` + ``uses:`` names a ready-made operator by its fully-qualified Python import + path (e.g. ``airflow.providers.standard.operators.bash.BashOperator``). + ``run:`` holds the arguments of a code task you implement elsewhere (see + `Code tasks`_ below). A task may have one or the other. + +``with`` + Operator-specific arguments. These are what you would pass to the operator's + constructor. Read on to understand the value grammar you can use in these + arguments. + +Task-level keys + Scheduling and common attributes sit next to the body, distinct from + ``with:``. These include things like ``retries``, ``trigger_rule``, + ``queue``, etc. Any key that is not one of the structural keys (``id``, + ``uses``, ``run``, ``with``, ``needs``, ``extends``) is passed to the + operator type, which validates it. + +``needs`` + A list of upstream tasks (referenced by ``id``) that must run before this + one. This describes an explicit dependency edge; XCom references in + ``with:`` or ``run:`` are discovered automatically and do not need to be + repeated here. + + +Templates and extends +--------------------- + +Tasks often share arguments. You can define a named partial task under the +top-level ``templates:`` key, and pull it into a task with ``extends``: + +.. code-block:: yaml + + templates: + retryable: + retries: 2 + shell: + extends: [retryable] # a template can extend other templates + uses: airflow.providers.standard.operators.bash.BashOperator + with: + cwd: /opt/airflow/scripts + + tasks: + - id: extract + extends: [shell] # pull the template into this task + with: + bash_command: "extract.sh" + - id: load + extends: [shell] + with: + bash_command: "load.sh" + +Both tasks run a ``BashOperator`` with ``retries: 2`` and the shared ``cwd``, +and add only their own ``bash_command``. A template may set anything a task can, +and may itself ``extends`` other templates. + +The templates named in ``extends`` are applied left to right, and the task's own +inline fields are applied last. Merging is shallow: ``with:`` and ``run:`` are +combined one key at a time, while every other field is taken as a whole rather +than deep-merged. + +The assembled task is validated against the real operator, so a field the +operator does not accept is reported as an error. It is also an error if a +non-existent template is referenced. + + +Values are literal by default +----------------------------- + +A value under ``with:`` (or ``run:``) is taken literally, unless it is a +single-key object whose key is a reserved ``$``-marker. For example, +``"echo foo.sh"`` and ``"{{ not rendered }}"`` specifies exactly that string, +not a template, even if ``bash_command`` is listed on the operator as a template +field. + +Four markers opt a value into special handling: + +``{$t: "..."}`` is a template string + A Jinja template, rendered when the task runs. This is the only way a value + is templated: ``{$t: "{{ ds }}"}``, ``{$t: "{{ params.region }}"}``, + ``{$t: "{{ conn.my_db.host }}"}``. The long form ``{$template: "..."}`` is + also accepted. + +``{$x: id}`` references an upstream output + The XCom output of an upstream task. ``{$x: extract}`` references + ``extract``'s return value, and ``{$x: {task: extract, key: rows}}`` a + specific key. Binding a value this way also creates the dependency edge + implicitly, so a task that consumes another's output does not need to also + list it in ``needs:``. The long form is ``{$xcom: ...}``. + +``{$f: path}`` reads a file's contents + The contents of a file stored next to the document is read and used. Review Comment: Could this state the boundary? `FilesystemDagDefinition.sibling` resolves the path against the document's directory and rejects anything that ends up outside it (an absolute path, `..`, a symlink out). Subdirectories are fine, which "next to the document" doesn't quite say. ########## airflow-core/docs/authoring-and-scheduling/yaml-dags.rst: ########## @@ -0,0 +1,349 @@ + .. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +.. _yaml-dags: + +Writing Dags in YAML +==================== + +|experimental| + +.. versionadded:: 3.4.0 + +Airflow can build a Dag from a YAML document. One YAML document describes one +Dag, and the YAML Dag importer turns it into the same Dag structure that Airflow +runs like any other source. The tasks can be ordinary Airflow operators, so +every provider operator is available without writing code. + +A document is declarative. Everything in it is a value, and the shape of each +value decides what it means. Since the format round-trips through JSON, a +document may equally be written as JSON. + +The following is a example YAML file defining a Dag: + +.. code-block:: yaml + + $schema: https://airflow.apache.org/schemas/dag/2026-10-30.json + dag_id: daily_report + schedule: "0 3 * * *" + start_date: 2024-01-01 + + tasks: + - id: extract + uses: airflow.providers.standard.operators.bash.BashOperator + with: + bash_command: + $t: "extract.sh {{ ds }}" + + - id: publish + uses: airflow.providers.standard.operators.bash.BashOperator + needs: [extract] + with: + bash_command: "publish.sh" + +This Dag runs every day at 03:00, The first task ``extract`` renders the date +into its command. The second ``publish`` waits for ``extract`` through +``needs`` and runs a fixed command. + +Enabling the importer +--------------------- + +The YAML importer is opt-in. Register it in +:ref:`config:dag_processor__dag_importer_configs`: + +.. code-block:: ini + + [dag_processor] + dag_importer_configs = [{"classpath": "airflow.sdk.importers.yaml_importer.YamlDagImporter"}] + +The importer handles ``.yaml`` and ``.yml`` files that declare a Dag ``$schema`` +(see below). For per-bundle registration and other options, see +:doc:`../administration-and-deployment/dag-importers`. + +The shape of a document +----------------------- + +``$schema`` (required) + The format version the document is written against, given as the URL of a + published schema. The version it names selects the rules the document is + parsed with. ``$schema`` is also how Airflow recognizes the file + as a Dag; a file without it is silently skipped. Editors that understand Review Comment: This only holds in discovery safe mode. With `[core] dag_discovery_safe_mode = False`, `might_contain_dag` returns True for every file, so a `.yaml` without `$schema` (a values file, CI config in the bundle) shows up as an import error saying `$schema` is missing. The sniff also matches the canonical URL prefix, not only the key, so in safe mode a typo in the URL makes the Dag vanish with nothing but a debug log. Maybe "in the default safe mode, a file without a Dag `$schema` URL is skipped; otherwise it is reported as an import error". ########## airflow-core/docs/authoring-and-scheduling/yaml-dags.rst: ########## @@ -0,0 +1,349 @@ + .. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +.. _yaml-dags: + +Writing Dags in YAML +==================== + +|experimental| + +.. versionadded:: 3.4.0 + +Airflow can build a Dag from a YAML document. One YAML document describes one +Dag, and the YAML Dag importer turns it into the same Dag structure that Airflow +runs like any other source. The tasks can be ordinary Airflow operators, so +every provider operator is available without writing code. + +A document is declarative. Everything in it is a value, and the shape of each +value decides what it means. Since the format round-trips through JSON, a +document may equally be written as JSON. + +The following is a example YAML file defining a Dag: + +.. code-block:: yaml + + $schema: https://airflow.apache.org/schemas/dag/2026-10-30.json + dag_id: daily_report + schedule: "0 3 * * *" + start_date: 2024-01-01 + + tasks: + - id: extract + uses: airflow.providers.standard.operators.bash.BashOperator + with: + bash_command: + $t: "extract.sh {{ ds }}" + + - id: publish + uses: airflow.providers.standard.operators.bash.BashOperator + needs: [extract] + with: + bash_command: "publish.sh" + +This Dag runs every day at 03:00, The first task ``extract`` renders the date +into its command. The second ``publish`` waits for ``extract`` through +``needs`` and runs a fixed command. + +Enabling the importer +--------------------- + +The YAML importer is opt-in. Register it in +:ref:`config:dag_processor__dag_importer_configs`: + +.. code-block:: ini + + [dag_processor] + dag_importer_configs = [{"classpath": "airflow.sdk.importers.yaml_importer.YamlDagImporter"}] + +The importer handles ``.yaml`` and ``.yml`` files that declare a Dag ``$schema`` +(see below). For per-bundle registration and other options, see +:doc:`../administration-and-deployment/dag-importers`. + +The shape of a document +----------------------- + +``$schema`` (required) + The format version the document is written against, given as the URL of a + published schema. The version it names selects the rules the document is + parsed with. ``$schema`` is also how Airflow recognizes the file + as a Dag; a file without it is silently skipped. Editors that understand + ``$schema`` will offer validation and completion with no extra setup. + +Dag-level keys + Attribute of the Dag sit at the top level, such as ``dag_id``, ``schedule``, + ``params``, etc. An unknown key is reported as an error. + +``tasks`` + A list of tasks. Each task has an ``id`` and exactly one body. + +Task body: ``uses`` or ``run`` + ``uses:`` names a ready-made operator by its fully-qualified Python import + path (e.g. ``airflow.providers.standard.operators.bash.BashOperator``). + ``run:`` holds the arguments of a code task you implement elsewhere (see + `Code tasks`_ below). A task may have one or the other. + +``with`` + Operator-specific arguments. These are what you would pass to the operator's + constructor. Read on to understand the value grammar you can use in these + arguments. + +Task-level keys + Scheduling and common attributes sit next to the body, distinct from + ``with:``. These include things like ``retries``, ``trigger_rule``, + ``queue``, etc. Any key that is not one of the structural keys (``id``, + ``uses``, ``run``, ``with``, ``needs``, ``extends``) is passed to the + operator type, which validates it. + +``needs`` + A list of upstream tasks (referenced by ``id``) that must run before this + one. This describes an explicit dependency edge; XCom references in + ``with:`` or ``run:`` are discovered automatically and do not need to be + repeated here. + + +Templates and extends +--------------------- + +Tasks often share arguments. You can define a named partial task under the +top-level ``templates:`` key, and pull it into a task with ``extends``: + +.. code-block:: yaml + + templates: + retryable: + retries: 2 + shell: + extends: [retryable] # a template can extend other templates + uses: airflow.providers.standard.operators.bash.BashOperator + with: + cwd: /opt/airflow/scripts + + tasks: + - id: extract + extends: [shell] # pull the template into this task + with: + bash_command: "extract.sh" + - id: load + extends: [shell] + with: + bash_command: "load.sh" + +Both tasks run a ``BashOperator`` with ``retries: 2`` and the shared ``cwd``, +and add only their own ``bash_command``. A template may set anything a task can, +and may itself ``extends`` other templates. + +The templates named in ``extends`` are applied left to right, and the task's own +inline fields are applied last. Merging is shallow: ``with:`` and ``run:`` are +combined one key at a time, while every other field is taken as a whole rather +than deep-merged. + +The assembled task is validated against the real operator, so a field the +operator does not accept is reported as an error. It is also an error if a +non-existent template is referenced. + + +Values are literal by default +----------------------------- + +A value under ``with:`` (or ``run:``) is taken literally, unless it is a +single-key object whose key is a reserved ``$``-marker. For example, +``"echo foo.sh"`` and ``"{{ not rendered }}"`` specifies exactly that string, +not a template, even if ``bash_command`` is listed on the operator as a template +field. + +Four markers opt a value into special handling: + +``{$t: "..."}`` is a template string + A Jinja template, rendered when the task runs. This is the only way a value + is templated: ``{$t: "{{ ds }}"}``, ``{$t: "{{ params.region }}"}``, + ``{$t: "{{ conn.my_db.host }}"}``. The long form ``{$template: "..."}`` is + also accepted. + +``{$x: id}`` references an upstream output + The XCom output of an upstream task. ``{$x: extract}`` references + ``extract``'s return value, and ``{$x: {task: extract, key: rows}}`` a + specific key. Binding a value this way also creates the dependency edge + implicitly, so a task that consumes another's output does not need to also + list it in ``needs:``. The long form is ``{$xcom: ...}``. + +``{$f: path}`` reads a file's contents + The contents of a file stored next to the document is read and used. + ``{$t: {$f: path}}`` allows reading the file and then rendering its contents + as a template. The long form is ``{$file: path}``. A file is loaded only + when you ask with ``$f``; it is never inferred from a filename suffix. + +``{$const: value}`` is a forced literal + An escape for the rare case that real data is shaped exactly like a marker. + ``{$const: {$x: 1}}`` is the literal dict ``{"$x": 1}``, not an XCom. + +Markers nest. A marker inside a list, or inside a nested mapping such as an +``env`` dict, is resolved by the same rules, so references and templates compose +through structure: + +.. code-block:: yaml + + with: + env: + RUN_DATE: + $t: "{{ ds }}" # rendered + REGION: us-east-1 # literal + +Dates and datetimes +------------------- + +A date or datetime argument accepts an ISO-8601 string or a YAML-native date, +and they mean the same thing: + +.. code-block:: yaml + + start_date: "2024-01-01" # ISO-8601 string + start_date: 2024-01-01 # YAML date + end_date: "2024-06-01T12:00:00" # ISO-8601 datetime string + +These are converted to a Python ``datetime`` before the Dag or task is built. A bare date becomes +midnight, and an ISO-8601 string is parsed. + +The conversion is applied only to an argument that expects a datetime. This is +decided from the operator's or Dag's argument **type annotation**. + +Using your own operators +------------------------ + +``uses:`` takes any operator's fully-qualified Python import path, so your own +operator class can work exactly like a provider's. + +.. code-block:: yaml + + - id: ingest + uses: my_company.operators.IngestOperator + trigger_rule: all_done # base arguments at the task-level + with: # your operator's own arguments + source: s3://bucket/incoming + cutoff: 2024-01-01 + +An operator takes two kinds of argument, kept in separate places. Your operator's own constructor +arguments go under ``with:``. The arguments every operator inherits from the base operator -- +``retries``, ``retry_delay``, ``trigger_rule``, ``queue``, ``pool``, ``priority_weight``, +``depends_on_past``, ``start_date``, and the rest of the task model -- go at the task level, beside Review Comment: Not everything in the task model can be set from YAML at this commit. `execution_timeout` needs a timedelta and there's no way to write one yet (`execution_timeout: 3600` fails BaseOperator's type check). Callbacks and `inlets`/`outlets` raise "not yet implemented" in models.py, and there's no `expand`. Since #74749 adds the duration syntax, should this land after it and document that here? A short "current limitations" list would also tell readers when they still need Python. ########## airflow-core/docs/administration-and-deployment/dag-importers.rst: ########## @@ -78,7 +82,7 @@ with: [dag_processor] dag_importer_configs = [ { - "classpath": "my_company.importers.YamlDagImporter", + "classpath": "airflow.sdk.importers.yaml_importer.YamlDagImporter", Review Comment: With this now pointing at the real class, the `my_company.importers.YamlDagImporter` placeholders on lines 91 and 104 (and in the `dag_importer_configs` description in config.yml) read like a second YAML importer you'd have to write yourself. Maybe switch those to the real classpath, or to a clearly custom name like `TomlDagImporter`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
