palakbhati opened a new pull request, #77293:
URL: https://github.com/apache/airflow/pull/77293

   closes: #77278
   
   With remote logging on, the triggerer uploaded each finished trigger's log 
inline in
   `TriggerRunnerSupervisor._handle_request`. A large batch (many deferrable 
sensors waiting on one
   event, or a restart with many timers already due) blocked the main loop for 
the whole batch. The
   runner waited for its reply, the runner watchdog read the silence as a 
deadlock and skipped the
   heartbeat, and with more than one triggerer the triggers were reassigned and 
could run twice.
   
   ### Changes
   - `finalize_trigger_log` pops the factory on the main thread and hands 
upload and close to a
     bounded `ThreadPoolExecutor`. New option `[triggerer] log_upload_workers` 
(default 4) also caps
     load on the remote store.
   - The remote log handler is loaded on the main thread before the pool 
starts, since
     `load_remote_log_handler()` initialises lazily and is not thread-safe.
   - Pending uploads are tracked by log path. Opening a trigger log waits up to 
10s for a pending
     upload of the same path, because opening truncates the file and the path 
is reused when a trigger
     is re-created or a task defers again in the same try.
   - `run()` drains pending uploads after `run_context` exits (so subclass 
teardown is unaffected),
     waits up to 10s, then cancels uploads that have not started and logs the 
counts.
   
   ### Notes for reviewers
   - `log_upload_timeout` is a fixed 10s class attribute, below the 30s health 
threshold. Happy to
     make it configurable.
   - The wait when opening a log runs on the main loop inside 
`_create_workload`'s DB session. It
     only happens when a path is reused while its upload is still running, and 
is capped at 10s.
   - `ThreadPoolExecutor` joins its workers at interpreter exit, so a hung 
upload can still delay
     process exit after `run()` returns. Remote clients normally have their own 
network timeouts.
   - `# noqa: SDK001` on the `load_remote_log_handler` import: the triggerer 
already depends on
     `airflow.sdk.log` for `upload_to_remote`.
   - Reassignment is covered indirectly: the tests check that the heartbeat is 
performed while
     every upload is still blocked. A real two-triggerer test would need a full 
deployment.
   
   ### Tests
   New tests in `test_triggerer_job.py` use event-blocked uploads instead of 
sleeps: heartbeat
   continues with 2,000 uploads blocked, concurrency never exceeds the worker 
count, a failed upload
   still closes its file, the handler loads on the main thread, re-creation and 
same-path triggers
   wait for the pending upload, a stuck upload only delays by the timeout, and 
`run()` drains or
   cancels on exit. Three existing tests now drain the pool before asserting.
   
   ---
   
   ##### Was generative AI tooling used to co-author this PR?
   
   - [X] Yes: Claude Code
   
   Generated-by: Claude Code following [the 
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to