alkismavridis commented on issue #57487: URL: https://github.com/apache/airflow/issues/57487#issuecomment-3461709202
@wjddn279 sure a low parallelism is an option, but the downside is that (if I understand this correctly) that one cannot have more than 5 Tasks running simultaniously, right? This can be a serious limitation for us. Our processes can be very long running, which means they will occupy one of these 5 precious slots. On the other hand, these long running tasks require very low processing power and RAM (they are just SSH Operators). The hight memory consumption of the spawned processes (I assume and correct me if I am wrong) does not come from the SSH Operator itself but from all the stuff around it: python interpreter, module imports, setting up global state etc. I hope there is a way we can spare having to do this setup 32 times just to have 32 lightweight tasks running in parallel. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
