We still use PGBouncer in Airflow 3, as we already used it for Airflow 2, but also because we use YugabyteDB, there we would need to install the Yugabyte specific Postgres driver if we would want to ditch PGBouncer, which we don't know what the effects would be as Yugabyte isn't (officially) supported on Airflow, but we are planning to test that in the future.
-----Original Message----- From: Jarek Potiuk <[email protected]> Sent: 23 October 2025 09:59 To: [email protected] Subject: Re: [DISCUSS] Using PGBouncer vs, SQL Alchemy pooling in Airflow 3 ? EXTERNAL MAIL: Indien je de afzender van deze e-mail niet kent en deze niet vertrouwt, klik niet op een link of open geen bijlages. Bij twijfel, stuur deze e-mail als bijlage naar [email protected]<mailto:[email protected]>. Thanks Jorrick - very interesting perspective.. I wonder if others have other experiences/thoughts they can share about it :) On Wed, Oct 22, 2025 at 11:50 PM Jorrick Sleijster <[email protected]> wrote: > Hey all, > > My two cents; > > Most likely, this is very specific to our setup, though it was good to > share nonetheless. > We use an internally managed HA Postgres cluster. Primary fail-overs > for this cluster are handled through DNS changes. > Furthermore, during maintenance on networking or data centers which > could impact the primary, we switch over the primary to another data > center as an extra precaution. > There is a slight difference in how the tools react: > - SQLAlchemy is reactive to DNS changes, waiting for a failure of a > connection before doing another DNS lookup. Furthermore, the > SQLAlchemy DNS lookup could potentially hit the DNS Cache of the > machine if the DNS record still has an active TTL. > - PGBouncer does (configurable) pro-active DNS lookups, thereby upon > fail-overs it is considerably faster to pick-up the new DNS name and > AIrflow never notices. > > Given we are not on Airflow 3 yet, I can not judge on the connection > pooling with a full picture and whether SQLAlchemy or PGBouncer would > be better. > What I can say is having connection pooling on in Airflow and > PGBouncer gave us quite some headachages with scaling & limits while > not having any benefits. > And although I'm stating the obvious; going from no connection pooling > to enabling connection pooling gave some db-heavy processes a 10x > performance boost on machines with a local Postgres database. > > Therefore, my opinion on the matter is that it depends on the database > setup, but the preferred setup should be *enable either of them, not both*. > > Kind regards, > Jorrick Sleijster > > > Op ma 20 okt 2025 om 20:30 schreef Jarek Potiuk <[email protected]>: > > > Hello Here, > > > > I wonder if people have some thoughts (experiences?) about using > PGBouncer > > for Airflow 3 ? People are asking questions - for example here > > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgi > > thub.com%2Fapache%2Fairflow%2Fdiscussions%2F56890&data=05%7C02%7Cdav > > id.blain%40infrabel.be%7C2cb1dfd0cfbb47e8ad3a08de120a1006%7Cb82bc314 > > ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C638968031591893364%7CUnknown%7CTW > > FpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMi > > IsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=fVqKPrQCopG5%2 > > BTSlaergm2N%2BiYssbKe%2BcK%2BlNqPHywY%3D&reserved=0 > > > > While in Airflow 2 PGBouncer was pretty much mandatory (due to > > workers using direct connections) - this is (likely) different story > > for Airflow > 3. > > I guess we should be able to (finally) - properly configure > > SQLAlchemy connection pools (how big? what's proper?) and PGBouncer > > might be simply unnecessary (leading to likely a bit less complex and > > faster deployment). > > > > Do people have thoughts / experience about it, I wonder? > > > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
