Hi Anand, >From my experience that reusing connection pool across those parsing processes would be dangerous( recall I got some connection corruption error because of that). If you find that to be too many connection you can probably tune down the max_threads <https://github.com/apache/incubator-airflow/blob/master/airflow/config_templates/default_airflow.cfg#L507> to reduce the number of parsing processes or make the scheduler parse slower by tune down min_file_process_interval <https://github.com/apache/incubator-airflow/blob/master/airflow/config_templates/default_airflow.cfg#L464>
Cheers, Kevin Y On Thu, Dec 6, 2018 at 5:53 PM anandgupta1...@gmail.com < anandgupta1...@gmail.com> wrote: > HI, > > I have a query. It will be great if somebody in the community can clarify > it. > > Basically, for each dag_processing process, I can see the orm being > reconfigured and disposed towards the end. This is creating and closing a > lot of connections on MySQL server, which crashes in scenarios where there > are a lot of new dags to process. > > I want to know if this is the intended behavior and why? Also what kind of > improvement can be planned in this area so that we can reuse the connection > pool amongst these processes. > > Thanks, > Anand >
