jscheffl commented on PR #56457: URL: https://github.com/apache/airflow/pull/56457#issuecomment-3756831222
> Overall, this is a significant improvement! thanks for implementing! Cool! Thanks for confirming my tests! Yes, this was my longer outstanding aim! But as LocalExecutor was also optimized via gc.freeze I assume memory can be optimized still down a bit. And with the limits on sockets... yeah in such a level of hundreds of tasks I assume some kernel parameters demand tuning as well. But for 400 concurrent tasks I assume we rather need to start thinking about supporting deferred mode on edge as well which would be a major next cool increment! Would it not be cool if the edge worker fetches the DeferredException and puts the triggerer actions into the async loop of the worker... almost zero effort and memory needed and then returns work back to normal supervisor when returnign from deferred state? Would need some changes on core or SDK as well... But yeah we need to have some dreams... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
