ucaeon opened a new pull request, #74659: URL: https://github.com/apache/airflow/pull/74659
`airflow db clean` can fail when cleaning up a stale Dag with task instances that have never started. For example, manually triggering a paused Dag can leave a queued Dag run and task instances with NULL start dates. If the Dag file is subsequently removed and the Dag becomes eligible for cleanup, the following can happen: 1. The task instances survive cleanup because their `start_date` is NULL. 2. The Dag is selected for deletion based on its `last_parsed_time`. 3. Deleting the Dag cascades to its `dag_version` rows, including versions still referenced by the surviving task instances. 4. The database rejects the deletion with a foreign-key constraint violation. Protecting referenced versions during `dag_version` cleanup alone is insufficient because deleting the parent Dag can still cascade to those versions. This change reuses the existing `skip_if_referenced` mechanism with `task_instance.dag_id` to exclude Dags that still have task instances. The reference check applies both when selecting cleanup candidates and immediately before deletion. Dags without surviving task instances remain eligible for cleanup. Regression tests verify that: - Referenced Dags, pinned versions, and never-started task instances are preserved, while unreferenced Dags and their versions are deleted. - Cleanup works with and without archiving, and with and without an explicit batch size. - A task instance added between candidate selection and deletion prevents its Dag from being deleted. All five new test cases fail with a foreign-key constraint violation without the fix. This PR addresses the cleanup failure in #71864. Deleting records with NULL start dates is a separate change tracked in #69872 and discussed in #69879. closes: #71864 ### Validation - `test_db_cleanup.py`: PostgreSQL 120 passed, MySQL 119 passed, SQLite 118 passed, with backend-specific skips. - Selective DB suites: API, Other, Core, CLI, Serialization, and Always passed on SQLite. - Selective non-DB suite: 4,988 passed. - Regular and manual static checks passed. - `apache-airflow` documentation build passed. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Codex (GPT-6) Generated-by: Codex (GPT-6) following [[the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
