dazza-codes edited a comment on issue #5788: [POC] multi-threading using asyncio URL: https://github.com/apache/airflow/pull/5788#issuecomment-569562651 Tagging this as related to - https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-28%3A+Add+AsyncExecutor+option - https://issues.apache.org/jira/browse/AIRFLOW-6395 👍 - I would like to see more done on this POC to create a new `AsyncExecutor` and apply it to async options for hooks, sensors and/or operators Once concern is that using `subprocess` might defeat the purpose of using coroutines for concurrency, if everything needs to be thrown across IPC anyway for this to work that way? That is, using subprocess incurs the overheads of multiprocessing, which should be avoided by async coroutines - right? I guess the point is that the event loop must start somewhere and this PR targets a new worker to manage an event loop? It just seems like the worker should be able to directly await an async callable (task/coroutine) without spawning it in a subprocess. (I guess it might make sense to rename this PR as `multiprocessing using asyncio`.)
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
