ashb commented on code in PR #1717: URL: https://github.com/apache/airflow-site/pull/1717#discussion_r4454289576
########## landing-pages/site/content/en/blog/airflow-3.4.0/index.md: ########## @@ -0,0 +1,513 @@ +--- +title: "Apache Airflow 3.4.0: Task Loops, Dags in Any Language, and AI Agents" +linkTitle: "Apache Airflow 3.4.0: Task Loops, Dags in Any Language, and AI Agents" +author: "Rahul Vats" +github: "vatsrahul1001" +linkedin: "vats-rahul" +description: "Apache Airflow 3.4.0 introduces bounded Task Loops for iterative workflows, Dag authoring beyond Python (TypeScript, Go, Java, and YAML) on the new AIP-85 Dag Importer, and a maturing common.ai provider for running LLM and agent tasks with built-in human approval." +tags: [Release] +date: "2026-11-04" +images: ["/blog/airflow-3.4.0/images/3.4.0.jpg"] +--- + +We're proud to announce the release of **Apache Airflow 3.4.0**! Where 3.3 gave your *tasks* memory and task bodies in multiple languages, 3.4 is about **shape and reach**. It brings **Task Loops**, a native loop primitive for workflows that repeat until they are done; the ability to author an entire Dag in TypeScript, Go, Java, or YAML rather than only Python; and a steadily maturing `common.ai` provider that turns LLM calls and agents into ordinary Airflow tasks, complete with human approval. + +**🎯 Release Highlights** + +📦 PyPI: https://pypi.org/project/apache-airflow/3.4.0/ \ +📚 Docs: https://airflow.apache.org/docs/apache-airflow/3.4.0/ \ +🛠️ Release Notes: https://airflow.apache.org/docs/apache-airflow/3.4.0/release_notes.html \ +🐳 Docker Image: `docker pull apache/airflow:3.4.0` \ +🚏 Constraints: https://github.com/apache/airflow/tree/constraints-3.4.0 + +# 🔁 Task Loops (AIP-111): Workflows That Repeat Until They Are Done + +Some workflows do not have a fixed shape. You walk a tree of unknown depth, you refine an estimate until the error is small enough, or you run an agentic *generate → score → refine* loop until the output is good. Until now you reached for hard-coding a fixed depth or for Dags that trigger themselves. Airflow 3.4 makes **bounded, sequential iteration** a native concept. + +Call `.loop()` on a decorated task group to repeat it a number of times decided at runtime, where each iteration can build on the result of the last. It behaves like a `do … while` loop with a hard cap: + +```python +from airflow.sdk import dag, task, task_group + + +@dag(schedule=None, catchup=False, tags=["example"]) +def refine_estimate(): + @task_group + def refine(): + @task + def improve(*, loop): + # First pass starts from 1.0; later passes read the previous result. + estimate = 1.0 if loop.index == 1 else loop.previous["estimate"] + return (estimate + 2.0 / estimate) / 2.0 + + @task + def evaluate(estimate): + return {"estimate": estimate, "error": abs(estimate * estimate - 2.0)} + + evaluate(improve()) + + def accurate_enough(*, loop): + return loop.result["error"] < 0.000001 + + @task + def finished(): + print("Refinement finished.") + + refinement = refine.loop(max_iterations=10, until=accurate_enough) + refinement >> finished() + + +refine_estimate() +``` + +## Key Capabilities + +- **`max_iterations`**: a required upper bound, so a loop can never run away. +- **`until`**: an optional predicate that decides when to stop. Return `True` to stop, or omit it entirely for a loop that runs a fixed number of times. Because the body runs before the check, it always runs at least once. +- **A `loop` context object** inside the body and the predicate: `loop.index` (the 1-based current pass), `loop.previous` (the last iteration's result), and `loop.result` (for the predicate). +- **Lazy materialization**: Airflow only creates the next iteration when the loop decides to continue, so you do not see a tail of skipped iterations. +- **Composable**: loops work with `.partial()` for body arguments, `.override()` for group config, and even `.expand()` to run mapped tasks *inside* a loop. + +A loop with a fixed count (no `until`) always runs exactly `max_iterations` times and succeeds. If a loop reaches `max_iterations` while its `until` predicate is still `False`, meaning it never converged, the final iteration is marked failed, so a failure to converge surfaces as a real failure. The Airflow UI gains dedicated loop views so you can see each iteration's history at a glance. + +> A few limits to know: loops cannot be nested, a loop cannot sit inside a mapped task group, and a whole loop cannot itself be mapped. Review Comment: ```suggestion > A few limits to know: loops cannot be nested, a loop cannot sit inside a mapped task group (though mapped tasks inside loops are okay), and a whole loop cannot itself be mapped. ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
