hussein-awala opened a new pull request, #36504: URL: https://github.com/apache/airflow/pull/36504
Airflow Metadata is the brain of Airflow; it is the central component used to store the state of all operations and enable communication between other components. A lot of the execution time of Airflow components is wasted waiting for some input of this Metadata, which we can avoid if we query the database asynchronously. We can perform other operations during the waiting time or send more requests to Metadata. SQLAlchemy supports since 1.4 [Asynchronous I/O (asyncio)](https://docs.sqlalchemy.org/en/14/orm/extensions/asyncio.html) that we can use to improve all our components: - improving scheduler logic - improving the async execution in the executors - making the Triggerer fully asynchronous - loading variables/connections asynchronously in the triggers and the async hooks. - and, of course, improving the performance of our API and webserver, which will serve much more users with lower resource usage. To test and validate the efficiency of async sessions before merging complicated methods from sync to async, I implemented the utils to create the async sessions and some async methods to load the variables and the connections from the database, and I tried one of the newly implemented methods with the triggerer: test_trigger.py: ```python from typing import Any from airflow.triggers.base import BaseTrigger class TestTrigger(BaseTrigger): def serialize(self) -> tuple[str, dict[str, Any]]: return "airflow.test_trigger.TestTrigger", {} async def run(self): import asyncio from airflow.models.connection import Connection for _ in range(1000): conn = await Connection.async_get_connection_from_secrets("airflow_db") self.log.info(conn.login) await asyncio.sleep(0.1) yield {} ``` dag.py: ```python from datetime import datetime from airflow.models.dag import DAG from airflow.models.operator import BaseOperator from airflow.test_trigger import TestTrigger class TestOperator(BaseOperator): def __init__(self, param: int, **kwargs): super().__init__(**kwargs) self.param = param def execute(self, context): self.defer( trigger=TestTrigger(), method_name="execute_complete", ) def execute_complete(self): self.log.info("Done!") with DAG( dag_id="async_db", schedule_interval=None, start_date=datetime(2023, 12, 31), catchup=False, ) as dag: TestOperator.partial(task_id="test_task").expand(param=list(range(1000))) ``` I triggered a manual Dag Run, which created 1000 mapped tasks and deferred them. I had the expected result in the log (login root) without any blocking in the event loop of the Triggerer, with reasonable resource consumption. Then I tested by replacing the trigger run method with: ```python async def run(self): import asyncio from airflow.models.connection import Connection for _ in range(1000): conn = Connection.get_connection_from_secrets("airflow_db") self.log.info(conn.login) await asyncio.sleep(0.1) yield {} ``` I got the result in the task log, but the Triggerer log was full of: ``` [2023-12-31T01:42:36.934+0000] {triggerer_job_runner.py:598} INFO - trigger async_db/manual__2023-12-31T01:42:07.645143+00:00/test_task/235/1 (ID 471) starting [2023-12-31T01:42:37.091+0000] {triggerer_job_runner.py:598} INFO - trigger async_db/manual__2023-12-31T01:42:07.645143+00:00/test_task/236/1 (ID 472) starting [2023-12-31T01:42:37.159+0000] {triggerer_job_runner.py:576} INFO - Triggerer's async thread was blocked for 0.57 seconds, likely by a badly-written trigger. Set PYTHONASYNCIODEBUG=1 to get more information on overrunning coroutines. [2023-12-31T01:42:37.369+0000] {triggerer_job_runner.py:598} INFO - trigger async_db/manual__2023-12-31T01:42:07.645143+00:00/test_task/237/1 (ID 473) starting [2023-12-31T01:42:37.549+0000] {triggerer_job_runner.py:598} INFO - trigger async_db/manual__2023-12-31T01:42:07.645143+00:00/test_task/239/1 (ID 474) starting [2023-12-31T01:42:37.772+0000] {triggerer_job_runner.py:576} INFO - Triggerer's async thread was blocked for 0.61 seconds, likely by a badly-written trigger. Set PYTHONASYNCIODEBUG=1 to get more information on overrunning coroutines. ``` and the Triggerer used almost all the resources of my computer. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
