Yes. Use pgbouncer. Look at "setup database" in our documentation and you will find out that this is the "production" setup with Postgres that we recommend.
On Fri, Feb 16, 2024 at 10:44 AM Abhay Sodani <sodaniab...@gmail.com> wrote: > > Hi Airflow Team, > > We are building an usecase in which we need to run 1000 parallel tasks in > a scheduled manner. > > For this we are planning to use Airflow with database as Postgres and > Celery as executor. > > While testing with 100 parallel tasks in Airflow, the observations is : > - 100 postgres connections are open and will remain open utill tasks get > completed. > > So assumption is each task is opening one connection and keeps connection > open untill task get completed. > > Now in order run 1000 parallel tasks, will it require 1000 postgres > connections? > > Is there any optimization or connection pooling for this. > > Please suggest. > > > Regards, > Abhay >
