Hey all, My two cents;
Most likely, this is very specific to our setup, though it was good to share nonetheless. We use an internally managed HA Postgres cluster. Primary fail-overs for this cluster are handled through DNS changes. Furthermore, during maintenance on networking or data centers which could impact the primary, we switch over the primary to another data center as an extra precaution. There is a slight difference in how the tools react: - SQLAlchemy is reactive to DNS changes, waiting for a failure of a connection before doing another DNS lookup. Furthermore, the SQLAlchemy DNS lookup could potentially hit the DNS Cache of the machine if the DNS record still has an active TTL. - PGBouncer does (configurable) pro-active DNS lookups, thereby upon fail-overs it is considerably faster to pick-up the new DNS name and AIrflow never notices. Given we are not on Airflow 3 yet, I can not judge on the connection pooling with a full picture and whether SQLAlchemy or PGBouncer would be better. What I can say is having connection pooling on in Airflow and PGBouncer gave us quite some headachages with scaling & limits while not having any benefits. And although I'm stating the obvious; going from no connection pooling to enabling connection pooling gave some db-heavy processes a 10x performance boost on machines with a local Postgres database. Therefore, my opinion on the matter is that it depends on the database setup, but the preferred setup should be *enable either of them, not both*. Kind regards, Jorrick Sleijster Op ma 20 okt 2025 om 20:30 schreef Jarek Potiuk <[email protected]>: > Hello Here, > > I wonder if people have some thoughts (experiences?) about using PGBouncer > for Airflow 3 ? People are asking questions - for example here > https://github.com/apache/airflow/discussions/56890 > > While in Airflow 2 PGBouncer was pretty much mandatory (due to workers > using direct connections) - this is (likely) different story for Airflow 3. > I guess we should be able to (finally) - properly configure SQLAlchemy > connection pools (how big? what's proper?) and PGBouncer might be simply > unnecessary (leading to likely a bit less complex and faster deployment). > > Do people have thoughts / experience about it, I wonder? >
