kaxil commented on code in PR #1486: URL: https://github.com/apache/airflow-site/pull/1486#discussion_r3044907430
########## landing-pages/site/content/en/blog/airflow-3.2.0/index.md: ########## @@ -0,0 +1,246 @@ +--- +title: "Apache Airflow 3.2.0: Data-Aware Workflows at Scale" +linkTitle: "Apache Airflow 3.2.0: Data-Aware Workflows at Scale" +author: "Rahul Vats" +github: "vatsrahul1001" +linkedin: "vats-rahul" +description: "Apache Airflow 3.2.0 introduces Asset partitioning for granular pipeline orchestration, multi-team deployments for enterprise scale, synchronous deadline alert callbacks, and continued progress toward full Task SDK separation." +tags: [Release] +date: "2026-04-07" +images: ["/blog/airflow-3.2.0/images/3.2.0.jpg"] +--- + +We're proud to announce the release of **Apache Airflow 3.2.0**! Airflow 3.1 puts humans at the center of automated workflows. 3.2 brings that same precision to data: Asset partitioning for granular pipeline orchestration, multi-team deployments for enterprise scale, synchronous deadline alert callbacks, and continued progress toward full Task SDK separation. + +**Details**: + +π¦ PyPI: https://pypi.org/project/apache-airflow/3.2.0/ \ +π Docs: https://airflow.apache.org/docs/apache-airflow/3.2.0/ \ +π οΈ Release Notes: https://airflow.apache.org/docs/apache-airflow/3.2.0/release_notes.html \ +π³ Docker Image: `docker pull apache/airflow:3.2.0` \ +π Constraints: https://github.com/apache/airflow/tree/constraints-3.2.0 + +# ποΈ Asset Partitioning (AIP-76): Only the Right Work Gets Triggered + +Asset partitioning has been one of the most requested additions to data-aware scheduling. If you work with date-partitioned S3 paths, Hive table partitions, BigQuery partitions, or really any partitioned data store, you've dealt with this: An upstream task updates one partition, and every downstream Dag fires regardless of which slice actually changed. It's wasteful, and for large deployments it creates real operational noise. + +Asset partitioning in 3.2 makes this granular. Downstream Dags trigger only when the specific partition they care about gets updated. It's the biggest change to data-aware scheduling since Assets were introduced, and it turns partition-driven orchestration into something Airflow handles natively rather than something you work around. + + + +## Key Capabilities + +* **Partition-driven scheduling**: Dags trigger on specific partition updates, not every asset change +* **CronPartitionTimetable**: Schedule Dags against partitions using cron expressions. Also available in the Task SDK +* **Backfill for partitioned Dags**: Backfill historical partitions without re-triggering everything downstream (#61464) +* **Multi-asset partitions**: A single Dag can listen for partitions across multiple assets, which matters when your downstream work depends on several sources aligning (#60577) + +For more advanced use cases, there are temporal and range partition mappers (#61522, #55247) for mapping time ranges and value ranges to partition keys, a partition key field on Dag run references (#61725) so you can inspect exactly which partition triggered a run, and PartitionedAssetTimetable for full control over how partition events from multiple assets get resolved into a unified trigger. + +**Example**: Three upstream ingestion Dags each write to a separate asset on an hourly cadence. The downstream Dag only triggers when all three have updated the same hourly partition. Since the three assets don't share a partition key natively, a mapper resolves them into a common key. + +```py +from __future__ import annotations + +from airflow.sdk import ( + DAG, + Asset, + CronPartitionTimetable, + PartitionedAssetTimetable, + StartOfHourMapper, + asset, + task, +) + +team_a_player_stats = Asset(uri="file://incoming/player-stats/team_a.csv", name="team_a_player_stats") +combined_player_stats = Asset(uri="file://curated/player-stats/combined.csv", name="combined_player_stats") + + +with DAG( + dag_id="ingest_team_a_player_stats", + schedule=CronPartitionTimetable("0 * * * *", timezone="UTC"), + tags=["player-stats", "ingestion"], +): + + @task(outlets=[team_a_player_stats]) + def ingest_team_a_stats(): + """Materialize Team A player statistics for the current hourly partition.""" + pass + + ingest_team_a_stats() + + +@asset(schedule=CronPartitionTimetable("15 * * * *", timezone="UTC")) +def team_b_player_stats(): + pass + + +with DAG( + dag_id="clean_and_combine_player_stats", + schedule=PartitionedAssetTimetable( + assets=team_a_player_stats & team_b_player_stats, + default_partition_mapper=StartOfHourMapper(), + ), + catchup=False, +): + + @task(outlets=[combined_player_stats]) + def combine_player_stats(dag_run=None): + """Merge the aligned hourly partitions into a combined dataset.""" + print(dag_run.partition_key) + + combine_player_stats() +``` + +See `example_asset_partition.py` and the Task SDK API docs for `PartitionedAssetTimetable` and partition mappers. + +# π’ Multi-Team Deployments (AIP-67): Airflow for the Enterprise + +Airflow 3.2 introduces multi-team support, allowing organizations to run multiple isolated teams within a single Airflow deployment. Each team can have its own Dags, connections, variables, pools, and executorsβ enabling true resource and permission isolation without requiring separate Airflow instances per team. + +This is particularly valuable for platform teams that serve multiple data engineering or data science teams from shared infrastructure, while maintaining strong boundaries between teams' resources and access. + +## Key Capabilities + +* **Per-team resource isolation**: Each team has its own Dags, connections, variables, and pools +* **Per-team executors**: Different teams can use different executors (e.g. Celery, Kubernetes, Local, AWS ECS, etc.) and configure them separately β #57837, #57910 +* **Team-scoped authorization**: Keycloak and Simple auth managers support team-scoped access control (#61351, #61861) +* **Team-scoped secrets**: Use `AIRFLOW_VAR__{TEAM}___{KEY}` environment variable or `AIRFLOW_CONN__<TEAM>___<CONN_ID>` pattern for team-specific secrets (#62588) +* **CLI management**: New CLI commands for managing teams (#55283) +* **UI team selector**: Team selector in connection, variable, and pool create/edit forms (#60237, #60474, #61082) +* **Full API support**: `team_name` field added to Connection, Variable, and Pool APIs (#59336, #57102, #60952) + +## Enabling Multi-Team + +``` +# In airflow.cfg: +[core] +multi_team = True + +# Or via environment variable: +export AIRFLOW__CORE__MULTI_TEAM=True +``` + +# β° Deadline Alerts: Now With Synchronous Callbacks (AIP-86) + +Building on the Deadline Alerts system introduced in Airflow 3.1, this release adds synchronous callback support. In 3.1, callbacks ran through the triggerer (async only), which limited integration options. Synchronous callbacks execute directly via the executor, with optional targeting of a specific executor via the executor parameter. + +## What's New in 3.2 + +* **SyncCallback support**: Callbacks now execute via the executor, not the triggerer, with optional executor targeting Review Comment: Weird wording.. SyncCallback execute on Worker, async does on triggerer -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
