GitHub user alkismavridis created a discussion: Thread-Powered Local Executor
### Description Implement a new executor type that executes tasks locally, but unlike LocalExecutor, it starts multiple threads instead of spawning multiple processes. ### Use case/motivation Our usecase is pretty straight forward. We use airflow to orchestrate our tasks that will run in different servers. To achieve that we use exclusively SSHOpeators. The actual heavy lifting of the task happens on the target machines (the tech stack of which may vary and is completely independent of airflow) At the moment we use LocalExecutor with a parallelism of 32. But unfotunately, the spawned processes of LocalExecutor eat way too much RAM, which causes significant memory issues on our infrastructure. As mentioned [here](https://github.com/apache/airflow/issues/56641#issuecomment-3460906655), each one of the 32 workers will eventually occupy about 120MB, and this memory will stay occupied even when the workers are idle. I think that in the case of SSHOperatos (and other super-lightweight tasks) the spawning of multiple processes can be an overkill. It would maybe make sense for such usecases to have a very simple executor that will run as a single process and will allocate a thread-pool of N threads instead of spawning N processes. If you think this is a good idea, will happily do an attempt to implement this myself and submit a PR. But I might need some support since I am not a python expert :) If you think that this is not a good idea, please suggest an alternative because for us having 5+ Gigabytes being allocated just to open couple of SSH Connections (aprox 120MB per connection) is a real pain. At the end of the day, my sole motivation is to be able to execute **lightweight tasks** without having to allocate the massive resources that LocalExecutor's workers consume, even when they are idle. Thanks :) ### Related issues https://github.com/apache/airflow/issues/56641#issuecomment-3460906655 ### Are you willing to submit a PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) GitHub link: https://github.com/apache/airflow/discussions/57699 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
