GitHub user brokenjacobs edited a comment on the discussion: Best practice for
webserver liveness probe check
Thanks for responding!
I managed to track this down to database pool size issues. I was only seeing
the timeouts when hitting the api server with terraform to set up
pools/variables, etc. And terraform uses parallel connections to do this. For
whatever reason this wasn't an issue on airflow 2.x but on 3.x I've had to
increase the number of connections I use to the db server. We use pg_bouncer,
so I had plenty available, but for whatever reason the api server was timing
out on getting a new connection to the DB with the default settings.
I just double checked. Specifically this was pgBouncer settings in the helm
chart:
pgbouncer:
metadataPoolSize: 40
resultBackendPoolSize: 20
It wasn't an apiserver setting but a pgbouncer setting. We need more
connections with airflow3 is the tldr;
GitHub link:
https://github.com/apache/airflow/discussions/54853#discussioncomment-14669029
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
