potiuk commented on issue #35267:
URL: https://github.com/apache/airflow/issues/35267#issuecomment-1786255118
I think indeed we need more details an investigation and answers (from your
side @a-meledin ) .
* Is it only with mapped tasks in that number you see the problem?
* Have you checked if you have enough resources to run your database with
this setup (I.e. handling 150 opened connections) ?
* Have you try to set lower/higher values? What happens them?
Generally speaking when you allow PGBouncer to allow 150 connections to your
database, then it will open all 150 connections and Postgres creates a separate
process for each connection. So if your postress does not have enough resources
(for example memory - but it can be CPU or other resources ) to run all your
processes, it will slow down to a crawl as those process will compete for those
resources.
You need to look at the resource usage and see what your bottleneck are.
Finally the last question and request:
* Did you see a regression comparing to other airflow versions or is it the
only version of Airflow you tested it with? Can you try with 2.7.1 and
upcoming (will be released tomorrow likely 2.7.3rc) and see if there are any
differences. There were some changes in 2.7.2 that **should** decrease the
number of connections used comparing to 2.7.1 but I wonder if this did not have
some side effects, so if you can check those, that would be great.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
