Re: [D] Dag Versioning keeps increasing [airflow]
GitHub user Timelessprod added a comment to the discussion: Dag Versioning keeps increasing I am having the same issue with a static DAG which has a `KubernetesPodOperator` where the image tag to use is provided with a variable in a config file which consumes an environment variable injected by Kubernetes. When I update this environment variable value in Kubernetes config and redeploy Airflow, it creates 20 to 70 new version tags for this DAG while the code of the DAG has not changed. I would expect only 1 new tag due to the value change. For me it's a bug at this point when I see this on the web UI: https://github.com/user-attachments/assets/6631524c-f698-43bb-acfe-a97a52695f8d"; /> GitHub link: https://github.com/apache/airflow/discussions/66103#discussioncomment-17824672 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
Re: [D] Dag Versioning keeps increasing [airflow]
GitHub user mateuscarestiato added a comment to the discussion: Dag Versioning keeps increasing Hi @ecodina! This is a known behavior in Airflow 3.x related to the new **DAG Versioning** system introduced in Airflow 3.0. Here's what's happening and how to address it: **Why versions keep increasing:** Airflow 3.x creates a new DAG version every time the DAG processor detects a **change in the DAG's serialized representation**. If your DAG is generated dynamically from a web form, even small non-deterministic elements can cause a "change" on every parse cycle. Common culprits: 1. **`datetime.now()` or `time.time()`** in the DAG definition (changes every parse) 2. **Random UUIDs or dynamic IDs** generated at import time 3. **Dict ordering inconsistencies** (Python 3.7+ guarantees insertion order, but external data sources may not) 4. **`pendulum.now()` used as `start_date`** without being fixed to a specific date **How to diagnose:** Run `dbt run` — wait, wrong tool 😄. In Airflow, compare two consecutive serialized DAG snapshots: ```bash airflow dags show # check if output changes between runs Or query the metadata DB directly: sql SELECT version_number, created_at, dag_code FROM dag_version WHERE dag_id = 'your_dag_id' ORDER BY created_at DESC LIMIT 5; Fix — make your dynamic DAG generation deterministic: python from datetime import datetime # ❌ Bad — changes on every parse start_date = datetime.now() # ✅ Good — fixed reference start_date = datetime(2024, 1, 1) For your web-form-generated DAGs specifically: Ensure the template renders identically for the same input (no timestamps injected at render time) Sort any dict/list structures before serialization Avoid id(object) or memory addresses in any string representations Let me know what your DAG generation code looks like and I can help narrow down the specific trigger! GitHub link: https://github.com/apache/airflow/discussions/66103#discussioncomment-16886822 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
Re: [D] Dag Versioning keeps increasing [airflow]
GitHub user wjddn279 added a comment to the discussion: Dag Versioning keeps increasing Yeah, I think that's right. I'm not sure why there's a package version mismatch between the dag-processors in the first place (since I don't know the your environment well), but it seems like the dag-processors are producing different results from each other and ping-ponging back and forth, causing the divergence. GitHub link: https://github.com/apache/airflow/discussions/66103#discussioncomment-16787494 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
Re: [D] Dag Versioning keeps increasing [airflow]
GitHub user ecodina edited a comment on the discussion: Dag Versioning keeps increasing Thanks! First of all, I ran this query: ```sql select dag_code.* from dag_code,dag_version where dag_code.dag_version_id =dag_version.id and dag_code.dag_id ='my_dag' order by dag_version.version_number; ``` What I saw is that the source_code_hash didn't change at all between versions: https://github.com/user-attachments/assets/7b3a28f4-84cb-46ef-9cf7-2ebdb223b099"; /> I then chose 2 versions (`019d6d4d-db14-78f1-ada6-300c326badd6` and `019d6d4e-d09d-7cdf-b97e-207399e21872`) that had been generated 2 minutes apart on the 8th April. With that, I saw that there was a difference in the `data` column for the task. One version had: ```json "template_fields":[ "command", "env", "slurm_options", "submit_as_user" ], ``` and the other: ```json "template_fields":[ "command", "env", "slurm_options", "submit_as_user", "cluster" ], cluster:"XXX" ``` The parameter "cluster" was added into the template_fields for the SlurmOperator available in our provider around mid March. We have 2 dag processor running, and I believe one of them is/was using an old version of the provider. This is weird, since we manage Docker Swarm using Portainer and redeploy using "Pull and Redeploy", which should restart all services and tasks. Does my hypothesis make sense? GitHub link: https://github.com/apache/airflow/discussions/66103#discussioncomment-16787368 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
Re: [D] Dag Versioning keeps increasing [airflow]
GitHub user ecodina added a comment to the discussion: Dag Versioning keeps increasing Thanks! First of all, I ran this query: ```sql select dag_code.* from dag_code,dag_version where dag_code.dag_version_id =dag_version.id and dag_code.dag_id ='my_dag' order by dag_version.version_number; ``` What I saw is that the source_code_hash didn't change at all between versions: https://github.com/user-attachments/assets/7b3a28f4-84cb-46ef-9cf7-2ebdb223b099"; /> I then chose 2 versions (`019d6d4d-db14-78f1-ada6-300c326badd6` and `019d6d4e-d09d-7cdf-b97e-207399e21872`) that had been generated 2 minutes apart on the 8th April. With that, I saw that there was a difference in the `data` column for the task. One version had: ```json "template_fields":[ "command", "env", "slurm_options", "submit_as_user" ], ``` and the other: ```json "template_fields":[ "command", "env", "slurm_options", "submit_as_user", "cluster" ], cluster:"XXX" ``` The parameter "cluster" was added into the template_fields for the SlurmOperator available in our provider. We have 2 dag processor running, and I believe one of them is/was using an old version of the provider. This is weird, since we manage Docker Swarm using Portainer and redeploy using "Pull and Redeploy", which should restart all services and tasks. Does my hypothesis make sense? GitHub link: https://github.com/apache/airflow/discussions/66103#discussioncomment-16787368 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
Re: [D] Dag Versioning keeps increasing [airflow]
GitHub user wjddn279 added a comment to the discussion: Dag Versioning keeps increasing @ecodina The most reliable way is to query the `serialized_dag` table in the metadata database and check how the `data` changes across versions. If you share the two versions where the change occurred, I should be able to help GitHub link: https://github.com/apache/airflow/discussions/66103#discussioncomment-16776787 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
Re: [D] Dag Versioning keeps increasing [airflow]
GitHub user potiuk added a comment to the discussion: Dag Versioning keeps increasing I think some parts of this could also be inernal representation of some of the Python objects. It could be connected to (say) base python verssion (lile 3.12.11 -> 3.12.12) :D GitHub link: https://github.com/apache/airflow/discussions/66103#discussioncomment-16774392 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
Re: [D] Dag Versioning keeps increasing [airflow]
GitHub user ecodina added a comment to the discussion: Dag Versioning keeps increasing Thanks Jarek! I'll try this out since it is quite a simple change, although I don't have a lot of hope since doing a `ls` inside the folder shows it was modified on February 11 and there are versions on April 16. GitHub link: https://github.com/apache/airflow/discussions/66103#discussioncomment-16765595 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
Re: [D] Dag Versioning keeps increasing [airflow]
GitHub user potiuk added a comment to the discussion: Dag Versioning keeps increasing My best guess is that the file is not generated atomically, and it's not fully written when parser parses it. Typical way of solving it is to write the files elsewhere and "mv" them. GitHub link: https://github.com/apache/airflow/discussions/66103#discussioncomment-16760055 This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
