potiuk opened a new pull request, #73551:
URL: https://github.com/apache/airflow/pull/73551
The per-job duration alert compares the median of the last three canary runs
against the baseline median. A job that was slow twice and has since
recovered
still clears that comparison, so the nightly report announces blips as
regressions.
The 2026-09-22 alert is the example: `MySQL 8.4:3.13 Core...Serialization`
was
reported at 25m 00s → 33m 17s (+33.2%), from the runs **25m · 33m · 34m** —
the
most recent run was already back at the baseline. On work time (the alert
excludes image build/pull) that job normally sits at 24–26m, so the two slow
nights were a blip, not a trend.
Per-job alerts now require every run in the latest window to clear the
relative
threshold on its own, not just their median (`JOB_REQUIRE_SUSTAINED`, on by
default). Replayed over the 17 successful canaries of 2026-09-12..21:
| as-of run | before | after |
|---|---|---|
| #45505 Sep 21 13:58 | 2 | 1 |
| #45385 Sep 21 02:03 | 2 | 0 |
| #45106 Sep 19 01:58 | 4 | 1 |
| #44410 Sep 16 01:59 | 4 | 3 |
| #44152 / #44027 / #43716 | 1 each | 0 |
16 job alerts over ten nights down to 5. What survives is the sustained
climbs —
the `constraints-version-check` jobs, whose runtime grows with every day of
dependency drift because each outdated package costs a full `uv sync` to
explain.
What goes are the DB-test and image-build one-nighters.
The report also carries the evidence behind each alert now, so it can be
judged
without opening the API — the flagged job's last runs, the band it usually
lands
in (the baseline's middle half, so one cache-cold night is not claimed as
"usual"), and the recent runs with start times and both image-excluded and
wall-clock durations:
```
*Jobs that got slower (image build excluded, slow on every recent run):*
• *Finalize tests / Deps 3.10:constraints-source-providers* — 17m 37s → *30m
34s* (+73.5%, usually 16m 01s–31m 59s)
_last runs (newest first): 31m · 33m · 27m · 16m · 17m · 18m · 18m · 23m_
*Last 8 runs (image build excluded; median 102m 24s, usually 89m 54s–124m
49s):*
• #45505 Sep 21 13:58 — 49m 54s (wall-clock 55m 35s)
• #45385 Sep 21 02:03 — 55m 02s (wall-clock 60m 40s)
…
```
The GitHub step summary gets the same detail; its recent-runs table now shows
start time and both durations instead of the `event` column, which is always
`schedule`.
Checks run: `scripts/tests/ci/test_analyze_ci_job_durations.py` (63 passed),
`prek run --from-ref main` pre-commit and manual stages (pass, including
zizmor).
No newsfragment — CI tooling, not user-facing.
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (please specify the tool below)
Generated-by: Claude Code (Opus 5) following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
---
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
