palakbhati opened a new pull request, #77293:
URL: https://github.com/apache/airflow/pull/77293
closes: #77278
With remote logging on, the triggerer uploaded each finished trigger's log
inline in
`TriggerRunnerSupervisor._handle_request`. A large batch (many deferrable
sensors waiting on one
event, or a restart with many timers already due) blocked the main loop for
the whole batch. The
runner waited for its reply, the runner watchdog read the silence as a
deadlock and skipped the
heartbeat, and with more than one triggerer the triggers were reassigned and
could run twice.
### Changes
- `finalize_trigger_log` pops the factory on the main thread and hands
upload and close to a
bounded `ThreadPoolExecutor`. New option `[triggerer] log_upload_workers`
(default 4) also caps
load on the remote store.
- The remote log handler is loaded on the main thread before the pool
starts, since
`load_remote_log_handler()` initialises lazily and is not thread-safe.
- Pending uploads are tracked by log path. Opening a trigger log waits up to
10s for a pending
upload of the same path, because opening truncates the file and the path
is reused when a trigger
is re-created or a task defers again in the same try.
- `run()` drains pending uploads after `run_context` exits (so subclass
teardown is unaffected),
waits up to 10s, then cancels uploads that have not started and logs the
counts.
### Notes for reviewers
- `log_upload_timeout` is a fixed 10s class attribute, below the 30s health
threshold. Happy to
make it configurable.
- The wait when opening a log runs on the main loop inside
`_create_workload`'s DB session. It
only happens when a path is reused while its upload is still running, and
is capped at 10s.
- `ThreadPoolExecutor` joins its workers at interpreter exit, so a hung
upload can still delay
process exit after `run()` returns. Remote clients normally have their own
network timeouts.
- `# noqa: SDK001` on the `load_remote_log_handler` import: the triggerer
already depends on
`airflow.sdk.log` for `upload_to_remote`.
- Reassignment is covered indirectly: the tests check that the heartbeat is
performed while
every upload is still blocked. A real two-triggerer test would need a full
deployment.
### Tests
New tests in `test_triggerer_job.py` use event-blocked uploads instead of
sleeps: heartbeat
continues with 2,000 uploads blocked, concurrency never exceeds the worker
count, a failed upload
still closes its file, the handler loads on the main thread, re-creation and
same-path triggers
wait for the pending upload, a stuck upload only delays by the timeout, and
`run()` drains or
cancels on exit. Three existing tests now drain the pool before asserting.
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes: Claude Code
Generated-by: Claude Code following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]