sbrinkhorst commented on issue #58310: URL: https://github.com/apache/airflow/issues/58310#issuecomment-5193506705
We recently migrated to Airflow 3.2 (from an Airflow 2 version) and I was surprised by the high version number of some DAGs but didn't pay much attention to it. After migrating to 3.3 we found out that it is actually a problem: if you go back to older DagRuns it crashes our API pods because it goes out of memory. We think this because Airflow 3.3 fixes the issue of not correctly seeing the run in the grid view for older DAGs. We parameterize our DAGs heavily and run a lot, so going back to an older run of a DAG that has 74k runs and 54k versions killed the pods. And lo and behold, it is indeed only for DAGs that have some random element in it. Did you do something to clean up the table, or should it be enough that we have database cleanup configured? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
