GitHub user mateuscarestiato added a comment to the discussion: Dag Versioning keeps increasing
Hi @ecodina! This is a known behavior in Airflow 3.x related to the new **DAG Versioning** system introduced in Airflow 3.0. Here's what's happening and how to address it: **Why versions keep increasing:** Airflow 3.x creates a new DAG version every time the DAG processor detects a **change in the DAG's serialized representation**. If your DAG is generated dynamically from a web form, even small non-deterministic elements can cause a "change" on every parse cycle. Common culprits: 1. **`datetime.now()` or `time.time()`** in the DAG definition (changes every parse) 2. **Random UUIDs or dynamic IDs** generated at import time 3. **Dict ordering inconsistencies** (Python 3.7+ guarantees insertion order, but external data sources may not) 4. **`pendulum.now()` used as `start_date`** without being fixed to a specific date **How to diagnose:** Run `dbt run` — wait, wrong tool 😄. In Airflow, compare two consecutive serialized DAG snapshots: ```bash airflow dags show <dag_id> # check if output changes between runs Or query the metadata DB directly: sql SELECT version_number, created_at, dag_code FROM dag_version WHERE dag_id = 'your_dag_id' ORDER BY created_at DESC LIMIT 5; Fix — make your dynamic DAG generation deterministic: python from datetime import datetime # ❌ Bad — changes on every parse start_date = datetime.now() # ✅ Good — fixed reference start_date = datetime(2024, 1, 1) For your web-form-generated DAGs specifically: Ensure the template renders identically for the same input (no timestamps injected at render time) Sort any dict/list structures before serialization Avoid id(object) or memory addresses in any string representations Let me know what your DAG generation code looks like and I can help narrow down the specific trigger! GitHub link: https://github.com/apache/airflow/discussions/66103#discussioncomment-16886822 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
