GitHub user wjddn279 added a comment to the discussion: Thread-Powered Local Executor
You can reduce the number of default worker processes (which is 32 by default) using the `AIRFLOW__CORE__PARALLELISM: 5` option. I am investigating other causes for the memory increase in the worker processes and will share my findings once I identify them. In my opinion, achieving parallelism through Python threads may not always behave as expected (since this is not typical concept of thread). In that case, the only option left is to create separate processes. GitHub link: https://github.com/apache/airflow/discussions/57699#discussioncomment-14848114 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
