kosteev opened a new issue, #26627: URL: https://github.com/apache/airflow/issues/26627
### Apache Airflow version Other Airflow 2 version ### What happened Airflow 2.3.3. Task instance failures produce duplicates in "task_fail" table for this constraint (dag_id, task_id, run_id, map_index). ### What you think should happen instead Recently FK constraint between task_fail and task_instance tables was introduced: https://github.com/apache/airflow/pull/22260 And then there was a change to purge duplicates for this constraint from task_fail table (on db upgrade): https://github.com/apache/airflow/pull/22769 Removing duplicates on db upgrade to Airflow 2.3+ before establishing FK between tables makes sense, however these duplicates can occur in running Airflow 2.3+ instance (see "How to reproduce"). What is rationale for removing duplicates once on upgrading to Airflow 2.3+ but keeping this possibility to generate duplicates again? Isn’t it going to break foreign key and integrity of these two tables? ### How to reproduce Trigger DAG with task that fails multiple times for different tries, it will produce duplications in "task_fail" table. Example (two tries with 5 mins interval for retries): ``` id | task_id | dag_id | start_date | end_date | duration | map_index | run_id 1 | task | dag1_failing | 2022-09-23 09:11:44.102894+00 | 2022-09-23 09:11:44.469007+00 | 0 | -1 | scheduled__2022-09-22T00:00:00+00:00 3 | task | dag1_failing | 2022-09-23 09:16:44.995269+00 | 2022-09-23 09:16:45.310398+00 | 0 | -1 | scheduled__2022-09-22T00:00:00+00:00 ``` ### Operating System Linux ### Versions of Apache Airflow Providers _No response_ ### Deployment Composer ### Deployment details _No response_ ### Anything else _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
