potiuk opened a new pull request, #69113: URL: https://github.com/apache/airflow/pull/69113
The daily **CI Duration Trend Alert** (`.github/workflows/ci-duration-monitor.yml` + `scripts/ci/analyze_ci_job_durations.py`) has been firing most nights with a wildly varying set of "jobs that got slower", carrying little real signal. **Root cause:** the monitor compared a **single** nightly canary run against the median of the preceding ~24 runs (`LATEST_RUNS` defaulted to `1`). The two sides were asymmetric — a raw, unsmoothed point against a robust median — so any one unlucky run (slow PyPI, GitHub runner queue pressure, a cold cache) tripped the alert. Because a different run is "latest" each day, a different set of jobs got flagged each day. Network-bound constraint-resolution jobs (`Finalize tests / Deps 3.x:constraints`, `Generate constraints`, provider installs), which legitimately swing tens of minutes run-to-run, dominated nearly every alert and only needed to clear a 3-minute absolute floor. **Fix:** - `LATEST_RUNS: "3"` — compare the median of the last 3 nightly runs against the baseline so the two sides are symmetric and one unlucky run no longer trips it. - `JOB_MIN_ABS_INCREASE_MINUTES: "6"` — require a larger sustained absolute jump before flagging an individual job, so ordinary network variance on the long constraint jobs stops alerting. No script logic changes — the existing env knobs already support this, so the behaviour is config-only and the script's tests are unaffected. Genuine sustained regressions still trip the alert. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Claude Code (Opus 4.8) Generated-by: Claude Code (Opus 4.8) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
