GitHub user alkismavridis added a comment to the discussion: Thread-Powered Local Executor
@wjddn279 sure a low parallelism is an option, but the downside is (if I understand this correctly) that one cannot have more than 5 Tasks running simultaneously, right? This can be a serious limitation for us. Our processes can be very long running, which means they will occupy one of these 5 precious slots for a long time, and the rest will be queued for who-knows how long. On the other hand, these long running tasks require very low processing power and RAM (they are just SSH Operators). The hight memory consumption of the spawned processes (I assume and correct me if I am wrong) does not come from the SSH Operator itself but from all the stuff around it: python interpreter, module imports, setting up global state etc. I hope there is a way we can spare having to do this setup 32 times just to have 32 lightweight tasks running in parallel. But as said, I am not a python expert at all. So not sure how well can Python do multithreading GitHub link: https://github.com/apache/airflow/discussions/57699#discussioncomment-14848115 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
