GitHub user wjddn279 added a comment to the discussion: Thread-Powered Local Executor
@alkismavridis Yes, I strongly agree that the excessive memory usage compared to the simplicity of the tasks stems from Python’s fork-based process creation model. If the memory issue cannot be resolved (I’m currently looking into it), switching to a model where workers are created dynamically—similar to the KubernetesExecutor—might be a good alternative. However, it’s hard to predict what impact that change might have, so I think we should get input from whoever is managing the overall direction. GitHub link: https://github.com/apache/airflow/discussions/57699#discussioncomment-14848116 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
