seanghaeli opened a new pull request, #76527:
URL: https://github.com/apache/airflow/pull/76527
Building on #76260, which added `/triggers`. An asset watcher has nowhere to
log: its lines only reach the triggerer's stdout and are lost when the
triggerer goes away (deferred tasks and callbacks already have logs covered).
This PR gives each watcher its own log file, uploaded with remote logging. A
follow-up PR will add `GET /triggers/{trigger_id}/logs` to read any trigger's
logs.
- Each watcher gets its own file on its triggerer, `triggers/<trigger
id>/<triggerer job id>/trigger.log`, with `trigger_id` on every line. The file
outlives the watcher's restarts. The lines still go to stdout by default
(`[triggerer] watcher_logs_to_stdout`).
- Remote logging uploads the file when the watcher leaves the triggerer or
the triggerer stops, and each rollover as it's made, so a busy watcher's whole
history reaches the store. A triggerer that starts uploads what a killed one
left behind, claiming it first so two triggerers starting together don't both
upload it.
- Disk use and open files are capped (`[triggerer] trigger_log_max_bytes`,
`trigger_log_max_open_files`). A full or read-only log disk doesn't stop the
triggerer. Without remote logging, or with a handler that keeps no copy of a
log without a task instance (Elasticsearch, OpenSearch), only the newest copy
of each watcher's file is kept, and the files of triggers that no longer exist
are removed at startup.
- Uploads run off the triggerer's main loop, so thousands of watchers or an
unreachable store don't stop it heartbeating. A watcher that comes back while
its last file is still uploading starts once that upload is done.
How to review: one commit. Start with `WatcherLoggingFactory` in
`jobs/triggerer_job_runner.py`, then `handle_orphaned_trigger_logs` for what
happens at startup. About 545 lines of source; the rest is tests and docs.
Breeze on Postgres, S3 remote logging on moto with `delete_local_logs` off,
`trigger_log_max_bytes` at 64 KiB: one watcher logging 5 lines a second, on a
triggerer that is stopped, started again, killed with `kill -9` and started a
third time on the same disk:
```text
# before (main)
=== trig-a's disk (75 s after start, 64 KiB cap):
no triggers/ directory
TO-CHATTY lines in trig-a's stdout: 465
=== S3 (75 s after the restart):
0 objects under logs/triggers
# after
=== trig-a's disk (75 s after start, 64 KiB cap):
triggers/1/3/rollover/trigger.log: 0 bytes, 0 lines, 0 with trigger_id=1
triggers/1/3/trigger.log: 46827 bytes, 121 lines, 121 with trigger_id=1
first line of the file:
{"timestamp":"2026-11-05T01:09:16.157865Z","level":"info","event":"TO-CHATTY
seq 340 xxxx...","trigger_id":1,...}
=== S3 (while trig-a runs: each rollover uploaded when made):
s3://to-logs/logs/triggers/1/3/rollover/trigger.log: 340 lines
=== S3 (after the graceful stop):
s3://to-logs/logs/triggers/1/3/rollover/trigger.log: 340 lines
s3://to-logs/logs/triggers/1/3/trigger.log: 133 lines
=== trig-a's disk (after the kill):
triggers/1/4/trigger.log: 44645 bytes, 116 lines, 116 with trigger_id=1
=== restart trig-a on the same disk
[info] Uploaded a trigger log left behind by a triggerer that is not alive
path=triggers/1/4/trigger.log triggerer_job_id=4
=== S3 (75 s after the restart):
s3://to-logs/logs/triggers/1/3/rollover/trigger.log: 340 lines
s3://to-logs/logs/triggers/1/3/trigger.log: 133 lines
s3://to-logs/logs/triggers/1/4/trigger.log: 116 lines
s3://to-logs/logs/triggers/1/5/rollover/trigger.log: 170 lines
```
A triggerer gets its 8 watchers back while their files are still uploading,
to a store that accepts connections and never answers (each upload takes about
90 s). Seconds since its last heartbeat, and how many watchers are assigned to
it:
```text
# before (the main loop waited for each upload)
t_s hb_age_s assigned_to_3
5 7 0
71 54 8
138 120 8
204 7 8
max heartbeat age over 200s: 168s; assigned back: 8 of 8
# after
t_s hb_age_s assigned_to_3
5 7 0
70 8 8
136 9 8
202 9 8
max heartbeat age over 200s: 10s; assigned back: 8 of 8
```
---
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR,
please
change below checkbox to `[X]` followed by the name of the tool, uncomment
the "Generated-by".
-->
- [X] Yes (please specify the tool below)
Generated-by: Claude Code (Opus 5.5) following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
