potiuk commented on PR #39484: URL: https://github.com/apache/airflow/pull/39484#issuecomment-2100975510
@hussein-awala > This pool executor is only used to execute short-running tasks, I don't think it will impact the memory usage, and I believe that this kind of objects (process/thread pool executor) should be created once and reused. > If it's really a risky change, we can add a new configuration to switch between the behaviors and change the default value later when the new behavior is tested in production. yes I think it's not **significant** overhead, I just wonder if there are no cases here which will accumulate over time. Actually in Python 3.11 there is a new feature in ProcessPoolExecutor that would be even better `max_tasks_per_child` that would be BEST approach - because then we could be able to run the process pool only once and let it recycle the processes after say 1000 tasks scheduled - and that would be rather safe change if we only do it in Python 3.11 and have a reasonable default for `max_tasks_per_child`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
