seanghaeli commented on issue #76209: URL: https://github.com/apache/airflow/issues/76209#issuecomment-6862193921
The triggerer's main loop does a full database pass every time the runner sends it anything, including a single log line, since `_service_subprocess(1)` returns as soon as any runner socket has data. Measured on main, one asset watcher logging 50 lines a second takes the triggerer from 1 to 51 passes a second, and from 7.5 to about 358 SQL statements a second (4 of every 7 row-locking). Details and numbers in #77061. Syncing the runner more often, as planned in this series, would wake the loop the same way, so it may be worth decoupling the database work from wake-ups first. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
