GitHub user potiuk added a comment to the discussion: Should Airflow keep SQLAlchemy connection pooling enabled when using PgBouncer (transaction pooling mode)?
This is a very good question and I think we have no good answer for now - i.e. no evidences not enough benchmarks to give definite answer. Purely teorethically - PGBouncer might not be needed any more and could be completely replaced by connection pooling in Airflow. PGBouncer used to be badly needed when each of the workers connnected directly to the DB, but since this is not happening in Airfow 3 - only scheduler, triggerer, api-server, dag processor - and you can limit a number of connections by using the SQLAlchemy pools you mentioned. So theorethically you **could** just skip PGBouncer and connect directly to the DB if you use SQLAlchemy pools. That would lkely give a slight boost over using PGBouncer. But I have no evidences for it. GitHub link: https://github.com/apache/airflow/discussions/56890#discussioncomment-14730990 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
