MatrixManAtYrService opened a new issue, #25210:
URL: https://github.com/apache/airflow/issues/25210
### Apache Airflow version
main (development)
### What happened
I have 16 dags which all update the same dataset. They're set to finish at
the same time (when the seconds on the clock are 00). About three quarters of
them behave as expected, but the other quarter fails with errors like:
```
[2022-07-21, 06:06:00 UTC] {standard_task_runner.py:97} ERROR - Failed to
execute job 8 for task increment_source ((psycopg2.errors.UniqueViolation)
duplicate key value violates unique constraint "dataset_dag_run_queue_pkey"
DETAIL: Key (dataset_id, target_dag_id)=(1, simple_dataset_sink) already
exists.
[SQL: INSERT INTO dataset_dag_run_queue (dataset_id, target_dag_id,
created_at) VALUES (%(dataset_id)s, %(target_dag_id)s, %(created_at)s)]
[parameters: {'dataset_id': 1, 'target_dag_id': 'simple_dataset_sink',
'created_at': datetime.datetime(2022, 7, 21, 6, 6, 0, 131730,
tzinfo=Timezone('UTC'))}]
(Background on this error at: https://sqlalche.me/e/14/gkpj); 375)
```
I've prepaired a gist with the details:
https://gist.github.com/MatrixManAtYrService/b5e58be0949eab9180608d0760288d4d
### What you think should happen instead
All dags should succeed
### How to reproduce
See this gist:
https://gist.github.com/MatrixManAtYrService/b5e58be0949eab9180608d0760288d4d
Summary: Unpause all of the dags which we expect to collide, wait two
minutes. Some will have collided.
### Operating System
docker/debian
### Versions of Apache Airflow Providers
n/a
### Deployment
Astronomer
### Deployment details
`astro dev start` targeting commit: cff7d9194f549d801947f47dfce4b5d6870bfaaa
be sure to have `pause` in requirements.txt
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
