amr-noureldin opened a new issue #9718:
URL: https://github.com/apache/airflow/issues/9718
**Apache Airflow version**: 1.10.10
**Environment**:
PostgreSQL 9.4
- **OS** (e.g. from /etc/os-release): Fedora 28
- **Kernel** (e.g. `uname -a`): Linux 3.10.0-1127.el7.x86_64 #1 SMP Tue Feb
18 16:39:12 EST 2020 x86_64 x86_64 x86_64 GNU/Linux
**What happened**: Upon enabling `store_serialized_dags`, we noticed that
connections pile up on the database end that are never closed. Out of 2500 open
db connections, 2495 had running queries:
```
DELETE FROM rendered_task_instance_fields WHERE
rendered_task_instance_fields.dag_id = ‘my_dag_id’ AND
rendered_task_instance_fields.task_id = ‘my_task_id’ AND
(rendered_task_instance_fields.dag_id, rendered_task_ins
tance_fields.task_id, rendered_task_instance_fields.execution_date) NOT IN
(SELECT rendered_task_instance_fields.dag_id,
rendered_task_instance_fields.task_id,
rendered_task_instance_fields.execution_date FROM rendered_task_instance_fields
WHERE rendered_task_instance_fields.dag_id = 'my_dag_id' AND
rendered_task_instance_fields.task_id = 'my_task_id' ORDER BY
rendered_task_instance_fields.execution_date DESC
LIMIT 30
```
Upon disabling `store_serialized_dags`, we no longer saw such a pile up of
db connections.
Irrelevant, but wouldn't hurt to mention: we are using pgbouncer as a
connection pool between Airflow and PostgreSQL 9.4. Pgbouncer is configured for
transactional connections.
**What you expected to happen**: I did not expect such queries to take that
long to execute, and block the db backend from further requests.
**How to reproduce it**:
- Enable store_serialized_dags
- Trigger a particularr DAG, with 3000 DagRuns
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
