seanghaeli opened a new issue, #77278:
URL: https://github.com/apache/airflow/issues/77278

   When many deferred tasks finish at once with remote logging on, the 
triggerer uploads each finished trigger's log inline in its main loop, one 
after the other (`TriggerRunnerSupervisor._handle_request`, 
`TriggerStateChanges`: `upload_to_remote()` then `close()` per finished 
trigger). While that runs it doesn't heartbeat or handle events. After a long 
batch it logs that its runner looks deadlocked and skips its heartbeat, and 
with more than one triggerer its other triggers are reassigned and run twice.
   
   This happens on ordinary workloads: many deferrable sensors waiting on one 
external event, or a restart after which many timers are already due.
   
   Suggestion: upload finished triggers' logs from a small thread pool instead 
of inline, so the main loop keeps heartbeating. The pool size also bounds how 
hard the remote store is hit.
   
   Done when: with remote logging on, a few thousand triggers finishing 
together don't stop the triggerer's heartbeat, their logs are all uploaded, and 
no trigger is reassigned, with a test.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to