GitHub user archsyscall created a discussion: Should Airflow keep SQLAlchemy connection pooling enabled when using PgBouncer (transaction pooling mode)?
Hi everyone, We’re running Airflow 3.1.0 (CeleryExecutor) on Kubernetes, with an external PostgreSQL (RDS) metadata database sitting behind PgBouncer configured in transaction pooling mode. The issue we’re seeing is that when triggering very large DAGs with many tasks, RDS Performance Insights shows high AAS (Average Active Sessions) and frequent Client:ClientWrite wait events. Interestingly, when we reduced the PgBouncer pool size, the AAS spikes decreased, and the database became noticeably more stable. This led us to wonder: Since PgBouncer is already handling connection pooling, does it still make sense for Airflow to maintain its own SQLAlchemy connection pool? In other words: When using PgBouncer in transaction pooling mode, is it better to disable Airflow’s internal SQLAlchemy pool (AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_ENABLED=False) and rely entirely on PgBouncer for connection management? Or does keeping a small SQLAlchemy pool still provide any meaningful benefit? GitHub link: https://github.com/apache/airflow/discussions/56890 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
