wjddn279 commented on issue #57487: URL: https://github.com/apache/airflow/issues/57487#issuecomment-3461961418
@alkismavridis Yes, I strongly agree that the excessive memory usage compared to the simplicity of the tasks stems from Python’s fork-based process creation model. If the memory issue cannot be resolved (I’m currently looking into it), switching to a model where workers are created dynamically—similar to the KubernetesExecutor—might be a good alternative. However, it’s hard to predict what impact that change might have, so I think we should get input from whoever is managing the overall direction. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
