alkismavridis opened a new issue, #57487: URL: https://github.com/apache/airflow/issues/57487
### Description Implement a new executor type that executes tasks locally, but unlike LocalExecutor, it starts multiple threads instead of spawning multiple processes ### Use case/motivation Our usecase is pretty straight forward. We use airflow to orchestrate our tasks that will run in different servers. To achieve that we use exclusively SSHOpeators. The actual heavy lifting of the task happens on the target machines (the tech stack of which may vary and is completely independent of airflow) At the moment we use LocalExecutor with a parallelism of 32. But unfotunately, the spawned processes of LocalExecutor eat way too much RAM, which causes significant memory issues on our infrastructure. As mentioned [here](https://github.com/apache/airflow/issues/56641#issuecomment-3460906655), each one of the 32 workers will eventually occupy about 200MB. I think that in the case of SSHOperatos (and other super-lightweight tasks that) the spawning of multiple processes can be an overkill. It would maybe make sense for such usecases to have a very simple executor that will run as a single process and will allocate a thread-pool of N threads instead of spawning N processes. If you think this is a good idea, will happily do an attempt to implement this myself and submit a PR. But I might need some support since I am not a python expert :) If you think that this is not a good idea, please suggest an alternative because for us having 5+ Gigabytes being allocated just to open couple of SSH Connections is a real pain. ### Related issues https://github.com/apache/airflow/issues/56641#issuecomment-3460906655 ### Are you willing to submit a PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
