GitHub user ricky26j added a comment to the discussion: Scaling Airflow 3 on EKS — API server OOMs, PgBouncer saturation, and health check flakiness at 8K concurrent tasks
@potiuk Thanks for the info. If PgBouncer is no longer recommended, does that mean this documentation is out of date? https://airflow.apache.org/docs/apache-airflow/stable/howto/set-up-database.html#setting-up-a-postgresql-database https://airflow.apache.org/docs/helm-chart/stable/production-guide.html#pgbouncer "Airflow is known - especially in high-performance setup - to open many connections to metadata database. This might cause problems for Postgres resource usage, because in Postgres, each connection creates a new process and it makes Postgres resource-hungry when a lot of connections are opened. Therefore we recommend to use [PGBouncer](https://www.pgbouncer.org/) as database proxy for all Postgres production installations." GitHub link: https://github.com/apache/airflow/discussions/62117#discussioncomment-17964586 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
