potiuk commented on issue #60144: URL: https://github.com/apache/airflow/issues/60144#issuecomment-3751605918
I would stronbly recommend those who look at gevent usage to see if they really want it. Since gevent is monkey-patching known libraries to make them compatible, it might easily fail when there are new libraries (like in this case) - when they are not yet supported by gevent, and there is not much Airflow team or Deployment Managers can do about it, until gevent team will implement suppport for those. Opening an issue in gevent or fastapi (byt I think due to the way how gevent work it's firmly responsibility of gevent team to make new releases that work with popular libraries like FastAPI) by those who see the need - is probably the fastest and most efficient way to signal gevent team that things should be fixed. Feel free to open an issue in gevent if not yet done @jrod9k1 or explore changing celery configuration to use another mechanism of parallelism. Gevent is really a poor (IMHO) way of achieving semi-parallel execution of I/O bound workloads using Greenlets. This indeed saves some memory in some cases, but it has a bunch of limitations - ESPECIALLY when dealing with newer async libraries (which FastAPI is an example of). I would recommend reading that blog post for example https://medium.com/@alpesh.ccet/the-power-of-green-deep-dive-into-pythons-gevent-for-high-performance-concurrency-d1d710dd20c6 -> and considering if you **really** need gevent - because it only makes sense if practically all your tasks are I/O bound - which generally speaking is anti-pattern in Airflow - deffered tasks if you need async operations are far better, native way of asyncIO waiting, and you are not really supposed to do heavy I/O operations in airflow tasks to justify gevent usage. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
