This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch backport-748148c-v3-4-test in repository https://gitbox.apache.org/repos/asf/airflow.git
commit f4351167204ce1deb35be452434861901619be21 Author: Jarek Potiuk <[email protected]> AuthorDate: Mon Nov 16 01:05:59 2026 +0100 [v3-4-test] Count passed fork CI jobs and post job statuses at the intended rate (#77125) * Post fork CI job statuses at the intended hourly rate GitHub starts the 5-minute monitor only every 9-10 minutes, so a fixed 13 writes a run posted about half the job statuses an hour the budget was sized for, and a few full-matrix fork runs at once took hours to show their jobs. A run that only deferred them also alerted Slack after nine runs, although each run made progress; a run whose statuses are still missing an hour after it finished is already reported on its own. Generated-by: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01NyQq9U7i5HhjDgrQ9qd3gq * Count passed fork jobs instead of posting a status for each A full-matrix fork run has about 130 jobs, and posting each passed one as its own commit status spent most of the shared write budget on what the aggregate status already says. Failures still get their own status, which is what a reviewer looks for. The time since the previous monitor run is read from GitHub, because the record carried between runs stays unchanged after a failed run or Slack post, which would have let the next run spend that budget again. Generated-by: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01NyQq9U7i5HhjDgrQ9qd3gq * Measure the job status budget between monitor run starts Counting from when reconciling starts credited each run's setup time again, and a re-run credited the time its first attempt already had, so the hourly budget could be exceeded. Generated-by: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01NyQq9U7i5HhjDgrQ9qd3gq * Count a monitor re-run from its previous attempt Counting from when the run was created credited each later attempt with the time earlier attempts already had. Generated-by: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01NyQq9U7i5HhjDgrQ9qd3gq * Keep the passed-jobs count when the fork status write fails The state comment records a completed run before the status is posted, and later runs do not read that run's jobs again, so a failed status write would have lost the count for good. Generated-by: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01NyQq9U7i5HhjDgrQ9qd3gq * Count cancelled fork jobs instead of posting a status for each A fork run cancelled by a newer push or a failing job cancels dozens of jobs at once, and each one became a red status that said no more than a count does, while spending the shared write budget. Generated-by: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01NyQq9U7i5HhjDgrQ9qd3gq * Report a fork run's cancelled jobs in one status The count in Fork CI / Tests (AMD) is easy to miss, while a status of its own shows in the pull request's checks that jobs were cancelled. A marker holding only the passed count now has its jobs read again, since it cannot give the cancelled count. Generated-by: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01NyQq9U7i5HhjDgrQ9qd3gq (cherry picked from commit 748148ccdb7f2383528627d9972f8f5f06a85d29) --- dev/AGENTS.md | 12 +- .../doc/ci/08_fork_ci_for_external_contributors.md | 105 +++-- .../src/airflow_breeze/utils/fork_ci_messages.py | 50 ++- .../src/airflow_breeze/utils/fork_ci_reconcile.py | 220 +++++++++-- dev/breeze/tests/test_fork_ci_messages.py | 66 ++++ dev/breeze/tests/test_fork_ci_reconcile.py | 427 ++++++++++++++++++--- 6 files changed, 746 insertions(+), 134 deletions(-) diff --git a/dev/AGENTS.md b/dev/AGENTS.md index c2ac0cbbb2f..66e0218a5ca 100644 --- a/dev/AGENTS.md +++ b/dev/AGENTS.md @@ -47,17 +47,17 @@ Document `uv run` (not `python`) as the invocation method in READMEs and instruc ## Fork CI job statuses and GitHub's rate limits The fork CI monitor (`breeze ci fork-ci-reconcile`, run by `.github/workflows/fork-ci-monitor.yml`) -reports every job of a gated pull request's fork run as its own commit status. Those writes are the +reports every failed job of a gated pull request's fork run as its own commit status, counts the +passed and cancelled jobs in `Fork CI / Tests (AMD)`, and reports the cancelled ones in one +`Fork CI / Cancelled jobs` status. Those writes are the largest share of what the monitor does with the repository's `GITHUB_TOKEN`, whose rate limits every workflow of `apache/airflow` shares (see "Write budget" in `dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md`). If the budget is exceeded, turn the job statuses off: set `job_statuses = false` in -`.github/fork-ci-gate.toml`. Signs that it is exceeded are the monitor's Slack note that job -statuses "have not fit in the budget" (posted only after job statuses were deferred for about 45 minutes in a row) -staying for hours, pull requests listed because their fork run finished more than an hour ago with -job statuses still not posted, its Slack note that GitHub rate-limited it for about 30 minutes in a row, the +`.github/fork-ci-gate.toml`. Signs that it is exceeded are pull requests listed because their fork +run finished more than an hour ago with job statuses still not posted, staying for hours, its Slack note that GitHub rate-limited it for about 30 minutes in a row, the "Rate limits left" line of its run log approaching zero, or other workflows of the repository failing on secondary rate limits. The aggregate `Fork CI / Tests (AMD)` status and `CI passed` are posted either way, so turning job statuses off loses detail, not gating. -Lowering `MAX_JOB_STATUSES_PER_RUN` in `fork_ci_reconcile.py` is the softer alternative. +Lowering `JOB_STATUSES_PER_HOUR` in `fork_ci_reconcile.py` is the softer alternative. diff --git a/dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md b/dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md index a1d932971a5..a9c7f42a1d2 100644 --- a/dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md +++ b/dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md @@ -886,7 +886,15 @@ and `gate_bypassed` (`error`) and `failed` (`failure`), yellow for `awaiting_pus when one exists and the fork's Actions page for `ci-amd.yml` otherwise, so a maintainer can open the fork run in one click; for `gate_bypassed` it is the project CI run that was observed. The `description` is a short sentence under 140 characters that always ends with -where to look next. +where to look next. For `running`, `failed`, `green` and `green_unverified` it is followed by the +fork run's passed and cancelled jobs, such as "Fork CI passed - see Details (127 jobs passed)" or +"Fork CI failed - see Details (97 jobs passed, 31 cancelled)", since neither is posted as statuses +of their own (see "Job statuses"). The counts are taken from the jobs the run read for its job +statuses, or else from the marker of a completed run whose jobs are all reported, so a status +write that failed after the state comment recorded the run still gets them; otherwise the status +keeps the counts it shows, and they are never carried over from another state. The reconciler +reads the state back from the description without the counts, and accepts counts only on those +four states. Statuses are immutable and per SHA. The reconciler compares the desired `(state, target_url, description)` with the current one fetched in the pass 2 query and posts only on @@ -900,8 +908,10 @@ monitor run rather than `details_url`. A status honours `target_url`. ### Job statuses -Next to the aggregate status, each job of the fork run is reported as its own commit status, so -the pull request shows which jobs run, failed or passed, each linking to the job in the fork. +Next to the aggregate status, each failed job of the fork run is reported as its own commit +status, so the pull request shows which jobs failed, each linking to the job in the fork. Passed +jobs are counted in `Fork CI / Tests (AMD)` instead, and cancelled jobs in it and in one status of +their own, `Fork CI / Cancelled jobs`. The run is the one `Fork CI / Tests (AMD)` reports (its `databaseId` is fetched with the check suites); its jobs come from `GET /repos/<fork>/actions/runs/<id>/jobs?filter=latest`. @@ -910,9 +920,20 @@ suites); its jobs come from `GET /repos/<fork>/actions/runs/<id>/jobs?filter=lat the prefix is what keeps a job from posting `Fork CI / Tests (AMD)`, from which the reconciler reads its decisions back, or `CI passed`, which the merge ruleset requires. Jobs that share a context, which GitHub compares case-insensitively, are reported as the worst of them. -- Only finished jobs are reported: a successful one as `success`, and a failed, timed-out, - cancelled or unstartable one as `failure`. Queued and running jobs are not, so each job costs one - write instead of two; `Fork CI / Tests (AMD)` shows the run as running, linking it, and with a +- Only finished jobs are reported: a failed, timed-out or unstartable one as `failure`, and a + successful one as `success` only over a status of the same job that is not a success, such as + its failure in an earlier run of the commit, which would otherwise keep showing. A full-matrix + run has about 130 jobs and each status is a write, so posting every passed job would take most + of the budget for what the aggregate status already says; on 2026-11-15 four such runs at once + took about six hours to show. +- Cancelled jobs get no status each: a run cancelled by a newer push or by a failing job cancels + dozens at once, and each would be a red status saying no more than a count does. Instead + `Fork CI / Cancelled jobs`, linking the fork run, is `failure` with "31 jobs cancelled in the + fork - see Details" while the run has cancelled jobs, and is overwritten with `success` and "No + jobs cancelled in the latest fork run" once a later run of the commit has none; it is not posted + for a run that never had any. The context is outside the job statuses' `Fork CI job /` prefix, so no job can + post it. A job the run cancelled keeps what an earlier run of the commit reported for it. Queued and running jobs are not + reported, so each job costs at most one write; `Fork CI / Tests (AMD)` shows the run as running, linking it, and with a 5-minute cadence a per-job "running" status was behind the fork's own run page anyway. A job of a re-run in the fork keeps the result of the earlier attempt until it finishes. Skipped and neutral jobs, and the jobs every run has (`Fork CI gate`, `Check if the run is needed`, @@ -921,33 +942,31 @@ suites); its jobs come from `GET /repos/<fork>/actions/runs/<id>/jobs?filter=lat (`GET /repos/apache/airflow/commits/<sha>/status`), and only differences are posted, failures first. GraphQL has no mutation for commit statuses (`createCheckRun` exists, but see above for why check runs do not fit), so each one is a REST call, and they count towards GitHub's limits on - content-creating requests: a reconciler run posts at most `MAX_JOB_STATUSES_PER_RUN` (13) of - them, each write charged as it is made. What does not fit is posted by the next run, and once - the budget is spent no further pull request's jobs or statuses are read in that run. One fork run - of the full matrix (about 130 jobs) needs more than a run's budget on its own, which the next - run catches up on, so deferring alone is not a problem: the run log lists the deferred pull - requests, and only when nine runs in a row deferred, about 45 minutes of the budget not - keeping up, is a note added to the Slack message. The count is kept in the record the monitor - carries between runs (`--reported-file`); without a readable record the note is added at once. - Its wording is fixed, so it is posted once while deferrals persist. A pull request whose fork run - finished more than an hour ago (the `updatedAt` of its completed check suite) and still has - statuses waiting is listed on its own as an attention item, so a delay that a busy backlog keeps - pushing back shows which pull request it holds up; the item names the run, so it is posted once - for that run. + content-creating requests: a reconciler run posts at most its share of `JOB_STATUSES_PER_HOUR` + (240) for the time since the previous run, each write charged as it is made (see "Write + budget"). What does not fit is posted by a later run, and once the budget is spent no further + pull request's jobs or statuses are read in that run. Deferring alone is not reported, since + later runs catch up and many failing runs at once can keep the monitor deferring while every + run makes progress: the run log lists the deferred pull requests. A pull request whose fork run finished more than an hour + ago (the `updatedAt` of its completed check suite) and still has statuses waiting is listed on + its own as an attention item, so a delay that a busy backlog keeps pushing back shows which pull + request it holds up; the item names the run, so it is posted once for that run. - Commit statuses cannot be deleted and have no neutral state. When a run has completed, a job context posted for an earlier run of the same commit that the completed run does not have, or skipped, is overwritten with `success` and the description "Not in the latest fork run - see - Fork CI / Tests (AMD)", so it no longer shows as running or failed. + Fork CI / Tests (AMD)", so it no longer shows as running or failed; one that is already a + success is left as it is. - Once every job of a completed run is reported, the run is recorded in the marker as - `jobs: "<run id>:<attempt>:<conclusion>"` (`runAttempt` is fetched with the check suites), so - later runs do not read its jobs again. A run in progress is read on every run, and a re-run in + `jobs: "<run id>:<attempt>:<conclusion>:<passed jobs>:<cancelled jobs>"` (`runAttempt` is + fetched with the check suites), so later runs do not read its jobs again. A marker recorded + before the counts were added has none, and its run's jobs are read once more. A run in progress is read on every run, and a re-run in the fork is read again because its attempt changes, even when it ends with the same conclusion. - Reporting job statuses is caught on its own: when it fails, the pull request is still classified, so for example a `CI passed` that no longer holds is still revoked. Reading a fork run's jobs fails now and then on GitHub server errors (HTTP 502, a reset HTTP/2 stream) that the next run gets past, so each failure is only logged, and the pull request is listed with the unknown items for Slack once its job statuses failed in nine runs in a row, about 45 minutes. The per pull request - count is kept in the same record as that of deferred job statuses; without a readable record it + count is kept in the record the monitor carries between runs (`--reported-file`); without a readable record it starts again. The item's wording is fixed whatever the error, so it is posted once while the failure persists, and the run log has each run's error. @@ -962,7 +981,7 @@ records this for agents. Each run logs what is left of the token's REST and GraphQL quotas (`GET /rate_limit`, which does not count against them) and how many of its calls GitHub rate-limited; the client retries those with backoff. Rate limiting in six runs in a row, about 30 minutes, adds a note to the Slack message, with fixed wording so -it is posted once while it persists; the count is kept in the same record as that of deferred job +it is posted once while it persists; the count is kept in the same record as that of failing job statuses. #### Write budget @@ -979,7 +998,10 @@ jobs whose conclusion is not `skipped`, minus the four jobs that are not reporte | CI change (full tests) | [40207881565](https://github.com/apache/airflow/actions/runs/40207881565) | 131 | 127 | `apache/airflow` had 1898 `Tests (AMD)` pull request runs from 2026-10-18 to 2026-10-24, about 270 a -day or 11 an hour. Each reported job costs one write, its result. +day or 11 an hour. Each reported job costs one write, its result. Since passed and cancelled jobs +are only counted in `Fork CI / Tests (AMD)`, a push costs one write per failed job (plus one per +job that passes after failing), and the estimates below, which assume every job is posted, are an +upper bound. The limits that matter are GitHub's documented ones for the `GITHUB_TOKEN` of `apache/airflow`, shared by every workflow of the repository: 1000 REST requests an hour, and 80 content-creating @@ -990,14 +1012,29 @@ requests a minute and 500 an hour. Assuming about 35 reported jobs for an averag monitor runs. - **Gate enabled**, assuming about half of the pull request runs come from gated pull requests (about 5-6 pushes an hour): about 200 writes an hour without a cap, 40% of the content-creating - limit and more at peaks. The cap of 13 writes a run, twelve runs an hour, holds job statuses to at - most 156 an hour: 31% of the content-creating limit, leaving the rest to the reconciler's other - writes (labels, comments, the aggregate statuses) and to the other workflows that share it. Each - run posts at most 13 statuses at about one a second, well under the 80-a-minute limit. Job - statuses then lag behind at busy times, the aggregate status does not. The `CI run started` - statuses of pull requests whose CI runs in `apache/airflow` come out of the same 13 writes a run, - first and at most 6 of them (see "The `CI run started` check"), so they do not raise the total; - at about 11 pull request runs an hour they take about 1 write a run. + limit and more at peaks. `JOB_STATUSES_PER_HOUR` holds job statuses to at most 240 an hour: 48% + of the content-creating limit, leaving the rest to the reconciler's other writes (labels, + comments, the aggregate statuses) and to the other workflows that share it. Job statuses then + lag behind at busy times, the aggregate status does not. The `CI run started` statuses of pull + requests whose CI runs in `apache/airflow` come out of the same budget, first and at most 6 a + run (see "The `CI run started` check"), so they do not raise the total; at about 11 pull + request runs an hour they take about 1 write a run. + +A run gets the hourly budget for the time between the previous monitor run's start and its own, +not one twelfth of it. Both are read from the monitor's runs (`GET +/repos/apache/airflow/actions/workflows/fork-ci-monitor.yml/runs`, runs still waiting in its +concurrency group left out), so they are the same boundary of each run, setup included, and not +from the record the monitor carries between runs, which a failed run or a failed Slack post leaves +unchanged although that run may have posted statuses. A re-run counts from its previous +attempt's start (`GET /repos/apache/airflow/actions/runs/<id>/attempts/<n>`), so the time earlier +attempts had is not credited again. GitHub +starts scheduled workflows later than their cron: on 2026-11-15 the 5-minute monitor ran every 9 +to 10 minutes, so a fixed 13 writes a run (160 an hour at the cron's cadence) posted only about 82 +an hour, and four full-matrix fork runs at once took about six hours to catch up. Runs queued +behind a slow one get little, since the time between them is short. At most +`MAX_JOB_STATUS_BUDGET_INTERVAL` (10 minutes) is counted, so a run after a gap, such as a +disabled workflow, posts at most 40 statuses at about one a second: half of the 80-a-minute +limit. When the previous run cannot be read, a run gets one cron interval's share, 20 writes. The per-run caps and the run counts before a Slack note are derived from `MONITOR_INTERVAL` in `fork_ci_reconcile.py`, which must match the monitor's cron (a test checks it), so changing the @@ -1238,7 +1275,7 @@ files, a jobs call and a compare call only per long `ci-amd.yml` run of a gated one changed-files call and one or two compare calls per head commit that turns green, plus one mutation per actual state change. Job statuses add a jobs call and a status call per pull request with a fork run in progress or a completed run not yet reported, and up to two status mutations per job, at most -`MAX_JOB_STATUSES_PER_RUN` per run; see "Write budget". Pull requests that pass 1 labelled and +the run's share of `JOB_STATUSES_PER_HOUR`; see "Write budget". Pull requests that pass 1 labelled and the search does not list yet add one GraphQL call per 25. Well inside the `GITHUB_TOKEN` limits at a 5-minute cadence. The reconciler logs its call counts so the budget stays observable. diff --git a/dev/breeze/src/airflow_breeze/utils/fork_ci_messages.py b/dev/breeze/src/airflow_breeze/utils/fork_ci_messages.py index 7378ad7f368..6ced6cf2600 100644 --- a/dev/breeze/src/airflow_breeze/utils/fork_ci_messages.py +++ b/dev/breeze/src/airflow_breeze/utils/fork_ci_messages.py @@ -37,9 +37,12 @@ from typing import NamedTuple STATUS_CONTEXT = "Fork CI / Tests (AMD)" # Each job of the fork run is reported under this prefix. Job names come from the contributor's -# workflow, so the prefix keeps them from posting STATUS_CONTEXT, CI_PASSED_CONTEXT or -# CI_RUN_STARTED_CONTEXT. +# workflow, so the prefix keeps them from posting STATUS_CONTEXT, CANCELLED_JOBS_CONTEXT, +# CI_PASSED_CONTEXT or CI_RUN_STARTED_CONTEXT. JOB_STATUS_CONTEXT_PREFIX = "Fork CI job / " +# The fork run's cancelled jobs, as one status rather than one each: a run cancelled by a newer push +# or a failing job cancels dozens of jobs at once. +CANCELLED_JOBS_CONTEXT = "Fork CI / Cancelled jobs" JOB_NAME_MAX_LENGTH = 100 # The check the "Require CI on main" ruleset requires before a pull request can (auto-)merge. Project # CI's gate job reports it; for a gated pull request, which never runs project CI, the reconciler @@ -120,7 +123,8 @@ class Marker: since: datetime # Set when the head commit's fork CI turned green and its base could be compared. base: BaseCheck | None = None - # The completed fork run (``<run id>:<attempt>:<conclusion>``) whose jobs are all reported. + # The completed fork run (``<run id>:<attempt>:<conclusion>:<passed jobs>``) whose jobs are all + # reported. jobs: str | None = None # The fork run (``<run id>:<attempt>``) whose failed jobs the comment lists. failed_run: str | None = None @@ -539,6 +543,16 @@ JOB_STATUSES = { STALE_JOB_STATUS = ("success", "Not in the latest fork run - see Fork CI / Tests (AMD)") +def cancelled_jobs_status(cancelled: int) -> tuple[str, str]: + """The ``(state, description)`` of the CANCELLED_JOBS_CONTEXT status for a fork run.""" + if cancelled: + return ( + "failure", + f"{cancelled} job{'' if cancelled == 1 else 's'} cancelled in the fork - see Details", + ) + return "success", "No jobs cancelled in the latest fork run" + + def job_status_context(job_name: str) -> str: # A line break or control character would make an unreadable check name. name = "".join(char if char.isprintable() else " " for char in job_name).strip() @@ -562,5 +576,31 @@ STATUS_STATES = { } -def status_description(state: str) -> str: - return _STATUS_DESCRIPTIONS[state] +class JobCounts(NamedTuple): + """The passed and cancelled jobs of a fork run, which are not posted as job statuses of their own.""" + + passed: int + cancelled: int = 0 + + +# States whose `Fork CI / Tests (AMD)` description counts the fork run's passed and cancelled jobs. +STATES_WITH_JOB_COUNTS = frozenset({"running", "failed", "green", "green_unverified"}) +_JOB_COUNTS_SUFFIX = re.compile(r" \((\d+) jobs? passed(?:, (\d+) cancelled)?\)\Z") + + +def status_description(state: str, job_counts: JobCounts | None = None) -> str: + description = _STATUS_DESCRIPTIONS[state] + if job_counts is None or state not in STATES_WITH_JOB_COUNTS: + return description + counts = f"{job_counts.passed} job{'' if job_counts.passed == 1 else 's'} passed" + if job_counts.cancelled: + counts += f", {job_counts.cancelled} cancelled" + return f"{description} ({counts})" + + +def split_job_counts(description: str) -> tuple[str, JobCounts | None]: + """A status description without its job counts, and the counts; None without them.""" + match = _JOB_COUNTS_SUFFIX.search(description) + if match is None: + return description, None + return description[: match.start()], JobCounts(int(match[1]), int(match[2] or 0)) diff --git a/dev/breeze/src/airflow_breeze/utils/fork_ci_reconcile.py b/dev/breeze/src/airflow_breeze/utils/fork_ci_reconcile.py index 61a30366219..65b6fd75170 100644 --- a/dev/breeze/src/airflow_breeze/utils/fork_ci_reconcile.py +++ b/dev/breeze/src/airflow_breeze/utils/fork_ci_reconcile.py @@ -58,6 +58,7 @@ from airflow_breeze.utils.fork_ci_gate import ( ) from airflow_breeze.utils.fork_ci_github import GitHubClient, GitHubError from airflow_breeze.utils.fork_ci_messages import ( + CANCELLED_JOBS_CONTEXT, CI_PASSED_CONTEXT, CI_PASSED_PROJECT_DESCRIPTION, CI_PASSED_REVOKED_DESCRIPTION, @@ -68,11 +69,14 @@ from airflow_breeze.utils.fork_ci_messages import ( JOB_STATUS_CONTEXT_PREFIX, JOB_STATUSES, STALE_JOB_STATUS, + STATES_WITH_JOB_COUNTS, STATUS_CONTEXT, STATUS_STATES, BaseCheck, FailedJob, + JobCounts, Marker, + cancelled_jobs_status, ci_passed_description, fork_actions_url, gate_modified_close_comment, @@ -83,6 +87,7 @@ from airflow_breeze.utils.fork_ci_messages import ( parse_marker, rebase_needed_description, released_comment, + split_job_counts, state_comment, status_description, strip_failed_job_list, @@ -199,9 +204,29 @@ UNREPORTED_JOB_NAMES = GATED_RUN_JOB_NAMES | {GATE_JOB_NAME, "Platform: AMD"} RUNNING_JOB_STATUSES = frozenset({"queued", "waiting", "pending", "requested", "in_progress"}) JOB_STATE_RANKS = {"failure": 0, "pending": 1, "success": 2} # Commit statuses count towards GitHub's limit of 500 content-creating requests per hour, shared by -# every workflow of the repository. This keeps job statuses to about a third of it; the rest follow -# on the next run, failures first. See "Job statuses" in the design doc. -MAX_JOB_STATUSES_PER_RUN = 160 // RUNS_PER_HOUR +# every workflow of the repository. This keeps job statuses under half of it; the rest follow on a +# later run, failures first. See "Write budget" in the design doc. +JOB_STATUSES_PER_HOUR = 240 +# GitHub starts scheduled runs later than their cron, the monitor often only every 9-10 minutes, so a +# run gets the hourly budget for the time since the previous run started rather than for +# MONITOR_INTERVAL. At +# most this long is counted, so a run after a gap (a disabled workflow, a lost record) stays at half +# of GitHub's limit of 80 content-creating requests a minute. +MAX_JOB_STATUS_BUDGET_INTERVAL = timedelta(minutes=10) +MONITOR_WORKFLOW_FILE = "fork-ci-monitor.yml" +# Enough to find the previous started run behind the runs queued in the monitor's concurrency group. +MONITOR_RUNS_READ = 10 +NOT_STARTED_RUN_STATUSES = frozenset({"queued", "waiting", "pending", "requested"}) + + +def compute_job_status_budget(since_previous_run: timedelta | None) -> int: + """The job status writes a run may make; without a previous run, those of one ``MONITOR_INTERVAL``.""" + if since_previous_run is None: + since_previous_run = MONITOR_INTERVAL + counted = min(max(since_previous_run, timedelta(0)), MAX_JOB_STATUS_BUDGET_INTERVAL) + return JOB_STATUSES_PER_HOUR * counted // timedelta(hours=1) + + # `CI run started` statuses of pull requests whose CI runs in apache/airflow come out of that budget # first: one write per new head commit, which merging waits for, while job statuses are informational. # At most this many, so job statuses keep the rest; the first runs after the check became required @@ -212,10 +237,6 @@ MAX_CI_RUN_STARTED_PER_RUN = 80 // RUNS_PER_HOUR MAX_PROJECT_CI_PASSED_PER_RUN = max(20 // RUNS_PER_HOUR, 1) # The GitHub Actions app, whose check suites are the workflow runs of a commit. GITHUB_ACTIONS_APP_ID = 15368 -# A single large fork run can need more writes than one run's budget, which the next run catches up -# on, so deferring is only reported when it lasts this many consecutive runs: then the budget does -# not keep up with the pull requests. -DEFERRED_RUNS_BEFORE_ALERT = count_runs_in(timedelta(minutes=45)) # GitHub answering with rate limits in this many consecutive runs is reported: a single burst is # retried, while a persisting one means the repository's shared limits are close to exhausted. RATE_LIMITED_RUNS_BEFORE_ALERT = count_runs_in(timedelta(minutes=30)) @@ -230,20 +251,14 @@ JOB_STATUSES_FAILED_REASON = ( "the run log has the errors" ) # A pull request whose fork run finished this long ago with job statuses still not all posted is -# reported by itself: deferring is expected to catch up within a few runs, not to leave one behind. +# reported. Deferring alone is not: several full-matrix fork runs at once (about 130 statuses each) +# keep later runs deferring for hours while every one of them makes progress. MIRROR_DELAY_BEFORE_ALERT = timedelta(hours=1) RATE_LIMITED_NOTE = ( f"GitHub rate-limited the fork CI monitor in {RATE_LIMITED_RUNS_BEFORE_ALERT} runs in a row; the " "repository's shared GITHUB_TOKEN limits are close to exhausted. Consider `job_statuses = false` in " ".github/fork-ci-gate.toml (see dev/AGENTS.md)" ) -# Fixed text, so the note is posted to Slack once while deferrals persist; the run log lists the pull -# requests. -JOB_STATUSES_DEFERRED_NOTE = ( - f"job statuses have not fit in the budget of {MAX_JOB_STATUSES_PER_RUN} writes per run for " - f"{DEFERRED_RUNS_BEFORE_ALERT} runs in a row and are posted by later runs; if this persists, set " - "`job_statuses = false` in .github/fork-ci-gate.toml (see dev/AGENTS.md)" -) RUNNING_STATUSES = frozenset({"QUEUED", "IN_PROGRESS", "PENDING", "WAITING", "REQUESTED"}) FAILED_CONCLUSIONS = frozenset({"FAILURE", "CANCELLED", "TIMED_OUT", "STARTUP_FAILURE", "ACTION_REQUIRED"}) @@ -601,20 +616,31 @@ def status_state(status: tuple[str, str, str] | None) -> str | None: """The state the reconciler recorded in a commit status it posted, if it posted it.""" if status is None: return None + description, job_counts = split_job_counts(status[2]) return next( ( state for state, github_state in STATUS_STATES.items() - if (github_state, status_description(state)) == (status[0], status[2]) + if (github_state, status_description(state)) == (status[0], description) + and (job_counts is None or state in STATES_WITH_JOB_COUNTS) ), None, ) def _set_status( - plan: Plan, pr: PullRequestSnapshot, state: str, target_url: str, *, force: bool = False + plan: Plan, + pr: PullRequestSnapshot, + state: str, + target_url: str, + *, + force: bool = False, + job_counts: JobCounts | None = None, ) -> None: - desired = (STATUS_STATES[state], target_url, status_description(state)) + """:param job_counts: the fork run's job counts; None keeps the counts the status already shows.""" + if job_counts is None and pr.current_status is not None and status_state(pr.current_status) == state: + job_counts = split_job_counts(pr.current_status[2])[1] + desired = (STATUS_STATES[state], target_url, status_description(state, job_counts)) if force or desired != pr.current_status: plan.status = desired @@ -666,6 +692,7 @@ def classify( find_gate_change: Callable[[], str | None] = lambda: None, find_failed_jobs: Callable[[], list[FailedJob] | None] = lambda: [], verify_head_repo_unavailable: Callable[[], bool] = lambda: False, + job_counts: JobCounts | None = None, ) -> Plan: """ Decide the mutations for one gated pull request. @@ -691,6 +718,8 @@ def classify( :param find_failed_jobs: the jobs of the fork run that did not pass, None when they cannot be read; only called for a failed run that started. On None the state comment is left for the next run, so that a failed read does not drop the list from it. + :param job_counts: the passed and cancelled jobs of the fork run, shown in `Fork CI / Tests + (AMD)`; None when they are not known, which keeps the counts the status shows. """ plan = Plan() labels = set(pr.labels) @@ -907,7 +936,12 @@ def classify( target_url = fork.run_url or fork_actions_url(pr.head_repo) startup_failure = state == ForkState.FAILED and fork.suite_conclusion == "STARTUP_FAILURE" _set_status( - plan, pr, "startup_failure" if startup_failure else state.value, target_url, force=refresh_status + plan, + pr, + "startup_failure" if startup_failure else state.value, + target_url, + force=refresh_status, + job_counts=job_counts, ) _set_ci_passed(plan, pr, state, base, target_url=target_url) _set_ci_run_started(plan, pr, fork) @@ -1849,6 +1883,39 @@ def fetch_fork_observations( return observations +def count_jobs(jobs: Iterable[dict[str, Any]]) -> JobCounts: + """The finished jobs of a fork run that passed and that were cancelled, of those job statuses cover.""" + conclusions = [ + job.get("conclusion") + for job in jobs + if job["name"] not in UNREPORTED_JOB_NAMES and job.get("status") not in RUNNING_JOB_STATUSES + ] + return JobCounts(conclusions.count("success"), conclusions.count("cancelled")) + + +def find_job_counts( + fork: ForkObservation, read_jobs: ReadJobs, jobs_reported: str | None +) -> JobCounts | None: + """ + The job counts of the fork run, from its jobs when this reconciler run read them, or else from + the key of the completed run whose jobs are all reported. + """ + if fork.run_id is None: + return None + jobs = read_jobs.get(fork.run_id) + if jobs is not None: + return count_jobs(jobs) + return parse_job_counts(jobs_reported) + + +def parse_job_counts(jobs_reported: str | None) -> JobCounts | None: + """The job counts in a completed run's key, ``<run id>:<attempt>:<conclusion>:<passed>:<cancelled>``.""" + parts = (jobs_reported or "").split(":") + if len(parts) != 5 or not (parts[3].isdigit() and parts[4].isdigit()): + return None + return JobCounts(int(parts[3]), int(parts[4])) + + def build_job_statuses(jobs: Iterable[dict[str, Any]]) -> dict[str, tuple[str, str, str]]: """The ``(state, target URL, description)`` to report for each finished fork job, by context.""" statuses: dict[str, tuple[str, str, str]] = {} @@ -1858,9 +1925,10 @@ def build_job_statuses(jobs: Iterable[dict[str, Any]]) -> dict[str, tuple[str, s if job["name"] in UNREPORTED_JOB_NAMES: continue outcome = "" if job.get("status") in RUNNING_JOB_STATUSES else job.get("conclusion") or "" - if outcome not in JOB_STATUSES: - # Unfinished jobs are left to `Fork CI / Tests (AMD)`; skipped and neutral ones did no work - # worth a check on the pull request. + if outcome not in JOB_STATUSES or outcome == "cancelled": + # Unfinished and cancelled jobs are left to `Fork CI / Tests (AMD)`, which counts the + # cancelled ones: a cancelled run cancels dozens at once. Skipped and neutral ones did no + # work worth a check on the pull request. continue state, description = JOB_STATUSES[outcome] context = contexts.setdefault( @@ -1928,7 +1996,7 @@ class JobStatusReporter: client: GitHubClient, *, repo: str, - budget: int = MAX_JOB_STATUSES_PER_RUN, + budget: int = compute_job_status_budget(None), now: datetime | None = None, read_jobs: ReadJobs | None = None, ): @@ -1968,7 +2036,7 @@ class JobStatusReporter: The key is recorded in the marker, so a completed run's jobs are read once. A run in progress, or one whose statuses did not all fit in the budget, is read again next time. The key holds the run attempt, so a re-run in the fork is read again even when it ends with the - same conclusion. + same conclusion, and the run's passed jobs, which later runs show without reading the jobs. """ if fork.run_id is None or not pr.head_repo: return None @@ -1978,8 +2046,14 @@ class JobStatusReporter: else None ) recorded = pr.marker.jobs if pr.marker is not None and pr.marker.sha == pr.head_sha else None - if key is not None and key == recorded: - return key + # Recorded before the job counts were, a key has none; its jobs are then read once more. + if ( + key is not None + and recorded is not None + and recorded.startswith(f"{key}:") + and parse_job_counts(recorded) is not None + ): + return recorded # Without writes left, the reads are spent for nothing: a backlog of hundreds of pull requests # would otherwise cost two reads each on every run. if self.remaining == 0: @@ -2004,12 +2078,29 @@ class JobStatusReporter: ) } desired = build_job_statuses(jobs) + counts = count_jobs(jobs) + cancelled_state, cancelled_description = cancelled_jobs_status(counts.cancelled) + desired[CANCELLED_JOBS_CONTEXT] = (cancelled_state, fork.run_url or "", cancelled_description) if key is not None: - # Jobs of an earlier run of this commit that the completed run did not run, or skipped. - wanted = {context.casefold() for context in desired} + # Jobs of an earlier run of this commit that the completed run did not run, or skipped. A + # cancelled job keeps what an earlier run reported for it. + wanted = {context.casefold() for context in desired} | { + job_status_context(job["name"]).casefold() + for job in jobs + if job.get("conclusion") == "cancelled" and job.get("status") not in RUNNING_JOB_STATUSES + } for folded, (context, (_, target_url, _)) in current.items(): if context.startswith(JOB_STATUS_CONTEXT_PREFIX) and folded not in wanted: desired[context] = (STALE_JOB_STATUS[0], target_url, STALE_JOB_STATUS[1]) + # Passed jobs are only counted in `Fork CI / Tests (AMD)`: a full-matrix run has about 130, and + # each status is a write. A success is posted only over a failure from an earlier run of the + # commit, which would otherwise keep showing; so is the cancelled jobs' one. + desired = { + context: status + for context, status in desired.items() + if status[0] != "success" + or current.get(context.casefold(), ("", ("success", "", "")))[1][0] != "success" + } changes = sorted( ( (context, status) @@ -2025,7 +2116,9 @@ class JobStatusReporter: self._client.set_status(pr.number, pr.head_sha, state, target_url, description, context=context) # Charged per write, so writes made before a later one fails still count. self.remaining -= 1 - return key + if key is None: + return None + return f"{key}:{counts.passed}:{counts.cancelled}" def fetch_workflow_state(client: GitHubClient, fork: str) -> str: @@ -2721,7 +2814,6 @@ def drop_reported( attention: list[tuple[int, str]], notes: list[str], record: bool = True, - deferred_runs: int = 0, rate_limited_runs: int = 0, job_status_failure_runs: dict[int, int] | None = None, ) -> tuple[list[tuple[int, str]], list[tuple[int, str]], list[str]]: @@ -2731,7 +2823,6 @@ def drop_reported( Only the previous run's items are kept, so an item that went away and comes back is posted again. A missing or unreadable file reports everything. - :param deferred_runs: the consecutive runs that deferred job statuses, recorded for the next run. :param rate_limited_runs: the consecutive runs GitHub rate-limited, recorded for the next run. :param job_status_failure_runs: per pull request, the consecutive runs whose job statuses failed, recorded for the next run. @@ -2740,7 +2831,6 @@ def drop_reported( "unknown": [list(item) for item in unknown], "attention": [list(item) for item in attention], "notes": notes, - "deferred_runs": deferred_runs, "rate_limited_runs": rate_limited_runs, "job_status_failure_runs": { str(number): runs for number, runs in (job_status_failure_runs or {}).items() @@ -2777,6 +2867,55 @@ def read_run_count(reported_file: Path | None, key: str) -> int | None: return count if isinstance(count, int) and not isinstance(count, bool) else 0 +def find_time_since_previous_monitor_run( + client: GitHubClient, repo: str, *, run_url: str +) -> timedelta | None: + """ + The time between the previous monitor run's start and this run's; None when it cannot be told. + + Both starts are read from GitHub, so they are the same boundary of each run, setup included, and a + failed run or a failed Slack post, which leave the record the monitor carries between runs + unchanged, still counts. A re-run of this run counts from its previous attempt's start. + """ + match = re.search(r"/actions/runs/(\d+)", run_url) + if match is None: + return None + run_id = int(match[1]) + try: + runs = client.rest_get( + f"repos/{repo}/actions/workflows/{MONITOR_WORKFLOW_FILE}/runs?per_page={MONITOR_RUNS_READ}" + )["workflow_runs"] + except Exception as e: + console_print(f"[warning]Could not read the monitor's earlier runs: {e}[/]") + return None + current = next((run for run in runs if run["id"] == run_id), None) + started = parse_github_datetime(current.get("run_started_at")) if current else None + if current is None or started is None: + return None + # Runs waiting in the concurrency group have not started, so they posted nothing. + boundaries = [ + start + for run in runs + if run["id"] != run_id and run.get("status") not in NOT_STARTED_RUN_STATUSES + for start in [parse_github_datetime(run.get("run_started_at"))] + if start is not None and start <= started + ] + attempt = current.get("run_attempt") or 1 + if attempt > 1: + try: + previous_attempt = client.rest_get(f"repos/{repo}/actions/runs/{run_id}/attempts/{attempt - 1}") + except Exception as e: + console_print(f"[warning]Could not read the previous attempt of this run: {e}[/]") + return None + previous_start = parse_github_datetime(previous_attempt.get("run_started_at")) + if previous_start is None: + return None + boundaries.append(previous_start) + if not boundaries: + return None + return started - max(boundaries) + + def read_job_status_failure_runs(reported_file: Path | None) -> dict[int, int]: """Per pull request, the consecutive runs whose job statuses failed, as the previous run recorded.""" if reported_file is None: @@ -2905,10 +3044,13 @@ def run_reconcile( f"listing the pull requests with `{LABEL_USE_PROJECT_CI}` failed: {type(e).__name__}" ) read_jobs: ReadJobs = {} + job_status_budget = compute_job_status_budget( + find_time_since_previous_monitor_run(client, repo, run_url=run_url) + ) job_reporter = JobStatusReporter( client, repo=repo, - budget=max(MAX_JOB_STATUSES_PER_RUN - ci_run_started_writes - ci_passed_reserved, 0), + budget=max(job_status_budget - ci_run_started_writes - ci_passed_reserved, 0), now=now, read_jobs=read_jobs, ) @@ -2964,6 +3106,7 @@ def run_reconcile( verify_head_repo_unavailable=functools.partial( check_head_repo_unavailable, client, repo, pr.number ), + job_counts=find_job_counts(fork, read_jobs, jobs_reported), ) # A revocation to `pending` always goes through: it is idempotent and must not depend on # `report_project_ci_passed` succeeding later in this run. @@ -3011,17 +3154,11 @@ def run_reconcile( attention.extend(job_reporter.delayed) notes = [bypass_finder.incomplete] if bypass_finder.incomplete else [] notes.extend(note for note in (ci_run_started_note, *ci_passed_notes) if note) - previous_deferred_runs = read_run_count(reported_file, "deferred_runs") - deferred_runs = (previous_deferred_runs or 0) + 1 if job_reporter.deferred else 0 if job_reporter.deferred: console_print( - f"[info]Job statuses of {len(job_reporter.deferred)} pull requests left for the next run " - f"({deferred_runs} runs in a row): {job_reporter.deferred}[/]" + f"[info]Job statuses of {len(job_reporter.deferred)} pull requests left for the next run, " + f"after a budget of {job_status_budget} writes: {job_reporter.deferred}[/]" ) - # Without a record of earlier runs, deferring is reported at once, as it cannot be told apart - # from a budget that does not keep up. - if previous_deferred_runs is None or deferred_runs >= DEFERRED_RUNS_BEFORE_ALERT: - notes.append(JOB_STATUSES_DEFERRED_NOTE) # The full list for the run log; Slack may show only part of a very long one. for number, reason in [*unknown, *attention]: console_print(f"[warning]#{number}: {reason}[/]") @@ -3042,7 +3179,6 @@ def run_reconcile( attention=attention, notes=notes, record=not client.dry_run, - deferred_runs=deferred_runs, rate_limited_runs=rate_limited_runs, job_status_failure_runs=job_status_failure_runs, ) diff --git a/dev/breeze/tests/test_fork_ci_messages.py b/dev/breeze/tests/test_fork_ci_messages.py index d945dfe3576..563af5b6723 100644 --- a/dev/breeze/tests/test_fork_ci_messages.py +++ b/dev/breeze/tests/test_fork_ci_messages.py @@ -38,9 +38,12 @@ from airflow_breeze.utils.fork_ci_messages import ( MAX_LISTED_FAILED_JOBS, OPEN_PR_LIMIT_URL, OVERLAP_LISTED, + STATES_WITH_JOB_COUNTS, BaseCheck, FailedJob, + JobCounts, Marker, + cancelled_jobs_status, ci_change_approval_comment, ci_modification_note_comment, ci_passed_description, @@ -53,7 +56,9 @@ from airflow_breeze.utils.fork_ci_messages import ( rebase_needed_description, released_comment, render_marker, + split_job_counts, state_comment, + status_description, strip_failed_job_list, tamper_close_comment, ungated_comment, @@ -298,10 +303,71 @@ def test_setup_texts_give_the_exact_command(text): assert "gh workflow enable ci-amd.yml -R u/fork" in text [email protected]( + ("state", "job_counts", "expected"), + [ + pytest.param("green", JobCounts(130), "Fork CI passed - see Details (130 jobs passed)", id="counted"), + pytest.param("running", JobCounts(1), "Fork CI running - see Details (1 job passed)", id="one-job"), + pytest.param( + "failed", + JobCounts(97, 31), + "Fork CI failed - see Details (97 jobs passed, 31 cancelled)", + id="with-cancelled", + ), + pytest.param("green", None, "Fork CI passed - see Details", id="not-counted"), + pytest.param( + "setup_required", + JobCounts(3, 1), + "Fork CI not set up - see the pull request comment", + id="state-without-a-run", + ), + ], +) +def test_status_description_counts_jobs(state, job_counts, expected): + assert status_description(state, job_counts) == expected + + [email protected]( + ("description", "expected"), + [ + pytest.param( + "Fork CI passed - see Details (130 jobs passed)", ("Fork CI passed - see Details", JobCounts(130)) + ), + pytest.param( + "Fork CI running - see Details (1 job passed)", ("Fork CI running - see Details", JobCounts(1)) + ), + pytest.param( + "Fork CI failed - see Details (97 jobs passed, 31 cancelled)", + ("Fork CI failed - see Details", JobCounts(97, 31)), + ), + pytest.param("Fork CI passed - see Details", ("Fork CI passed - see Details", None)), + pytest.param( + "Fork CI passed (2 jobs passed) - see Details", + ("Fork CI passed (2 jobs passed) - see Details", None), + ), + ], +) +def test_split_job_counts(description, expected): + assert split_job_counts(description) == expected + + [email protected]( + ("cancelled", "expected"), + [ + pytest.param(31, ("failure", "31 jobs cancelled in the fork - see Details"), id="cancelled"), + pytest.param(1, ("failure", "1 job cancelled in the fork - see Details"), id="one"), + pytest.param(0, ("success", "No jobs cancelled in the latest fork run"), id="none"), + ], +) +def test_cancelled_jobs_status(cancelled, expected): + assert cancelled_jobs_status(cancelled) == expected + + def test_status_descriptions_fit_github_limit(): base_ref = "providers-cncf-kubernetes/v10-22" descriptions = [ *_STATUS_DESCRIPTIONS.values(), + *(status_description(state, JobCounts(99999, 99999)) for state in STATES_WITH_JOB_COUNTS), CI_PASSED_REVOKED_DESCRIPTION, ci_passed_description(BaseCheck("d" * 40, 199), base_ref=base_ref), *( diff --git a/dev/breeze/tests/test_fork_ci_reconcile.py b/dev/breeze/tests/test_fork_ci_reconcile.py index 4e2ddd2bf90..55246b5c0a3 100644 --- a/dev/breeze/tests/test_fork_ci_reconcile.py +++ b/dev/breeze/tests/test_fork_ci_reconcile.py @@ -31,6 +31,7 @@ from airflow_breeze.commands.ci_commands import fork_ci_reconcile from airflow_breeze.utils.fork_ci_gate import GateConfig from airflow_breeze.utils.fork_ci_github import GitHubClient, GitHubError from airflow_breeze.utils.fork_ci_messages import ( + CANCELLED_JOBS_CONTEXT, CI_PASSED_CONTEXT, CI_PASSED_PROJECT_DESCRIPTION, CI_PASSED_WAITING_PROJECT_CI_DESCRIPTION, @@ -42,18 +43,17 @@ from airflow_breeze.utils.fork_ci_messages import ( STATUS_CONTEXT, BaseCheck, FailedJob, + JobCounts, Marker, parse_marker, render_marker, ) from airflow_breeze.utils.fork_ci_reconcile import ( - DEFERRED_RUNS_BEFORE_ALERT, GATE_PROTECTED_PATHS, JOB_STATUS_FAILURE_RUNS_BEFORE_ALERT, - JOB_STATUSES_DEFERRED_NOTE, JOB_STATUSES_FAILED_REASON, MAX_CI_RUN_STARTED_PER_RUN, - MAX_JOB_STATUSES_PER_RUN, + MAX_JOB_STATUS_BUDGET_INTERVAL, MAX_PROJECT_CI_PASSED_PER_RUN, MONITOR_INTERVAL, RATE_LIMITED_NOTE, @@ -77,7 +77,9 @@ from airflow_breeze.utils.fork_ci_reconcile import ( check_head_repo_unavailable, classify, compare_with_base, + compute_job_status_budget, compute_projection, + count_jobs, discover, drop_reported, fetch_fork_observations, @@ -86,10 +88,12 @@ from airflow_breeze.utils.fork_ci_reconcile import ( find_ci_run_since, find_failed_jobs, find_heads_shared_across_bases, + find_job_counts, find_project_ci_label_heads, find_project_ci_passed_url, find_project_ci_run_url, find_state_comment, + find_time_since_previous_monitor_run, is_cancelled_gate_run, is_gated_skip_run, is_own_pilot_change, @@ -108,6 +112,7 @@ from airflow_breeze.utils.fork_ci_reconcile import ( report_project_ci_passed, report_project_ci_runs_started, run_reconcile, + status_state, ) from airflow_breeze.utils.path_utils import AIRFLOW_ROOT_PATH @@ -655,6 +660,20 @@ def test_green_records_the_base_it_tested(pr_overrides, compared, expected_base) ), pytest.param(None, _marker("green"), ForkState.GREEN, True, id="marker-is-not-trusted"), pytest.param(("success", RUN_URL, "forged"), None, ForkState.GREEN, True, id="unknown-status"), + pytest.param( + ("success", RUN_URL, "Fork CI passed - see Details (130 jobs passed)"), + None, + ForkState.GREEN, + False, + id="status-green-with-passed-jobs", + ), + pytest.param( + (UNVERIFIED_STATUS[0], RUN_URL, f"{UNVERIFIED_STATUS[2]} (1 job passed)"), + None, + ForkState.GREEN_UNVERIFIED, + False, + id="status-unverified-with-passed-jobs", + ), ], ) def test_green_trust_is_decided_once_per_commit(current_status, marker, expected_state, lists_files): @@ -699,6 +718,96 @@ def test_green_is_not_trusted_for_a_head_shared_across_bases(): changes.assert_not_called() [email protected]( + ("status", "expected"), + [ + pytest.param( + ("success", RUN_URL, "Fork CI passed - see Details (12 jobs passed)"), "green", id="green" + ), + pytest.param( + ("pending", RUN_URL, "Fork CI running - see Details (1 job passed)"), "running", id="running" + ), + pytest.param( + ("error", RUN_URL, "Project CI ran for a gated pull request - see Details (3 jobs passed)"), + None, + id="count-on-a-state-without-one", + ), + pytest.param( + ("success", RUN_URL, "Fork CI passed - see Details (many jobs passed)"), None, id="no-number" + ), + pytest.param( + ("failure", RUN_URL, "Fork CI failed - see Details (97 jobs passed, 31 cancelled)"), + "failed", + id="with-cancelled", + ), + ], +) +def test_status_state_reads_job_counts(status, expected): + assert status_state(status) == expected + + [email protected]( + ("current_status", "job_counts", "expected_status"), + [ + pytest.param( + None, + JobCounts(130), + ("success", RUN_URL, "Fork CI passed - see Details (130 jobs passed)"), + id="counted", + ), + pytest.param( + ("success", RUN_URL, "Fork CI passed - see Details (130 jobs passed)"), + None, + None, + id="count-kept-when-jobs-not-read", + ), + pytest.param( + ("success", RUN_URL, "Fork CI passed - see Details (129 jobs passed)"), + JobCounts(130), + ("success", RUN_URL, "Fork CI passed - see Details (130 jobs passed)"), + id="count-updated", + ), + pytest.param( + ("success", RUN_URL, "Fork CI passed - see Details (130 jobs passed, 2 cancelled)"), + None, + None, + id="cancelled-count-kept", + ), + ], +) +def test_fork_status_shows_the_job_counts(current_status, job_counts, expected_status): + plan = classify( + _pr(current_status=current_status), + GREEN, + config=CONFIG, + now=NOW, + repo=REPO, + find_bypass_run=lambda: None, + changes_ci_files=lambda: False, + compare_base=lambda: CURRENT_BASE, + job_counts=job_counts, + ) + + assert plan.state == ForkState.GREEN + assert plan.status == expected_status + + +def test_fork_status_does_not_carry_a_count_into_another_state(): + running = ("pending", RUN_URL, "Fork CI running - see Details (40 jobs passed)") + plan = classify( + _pr(current_status=running), + GREEN, + config=CONFIG, + now=NOW, + repo=REPO, + find_bypass_run=lambda: None, + changes_ci_files=lambda: False, + compare_base=lambda: CURRENT_BASE, + ) + + assert plan.status == GREEN_STATUS + + def test_find_heads_shared_across_bases(): prs = [ _pr(number=1, head_sha="a", base_ref="main"), @@ -2790,7 +2899,6 @@ def test_drop_reported_reports_everything_without_a_readable_previous_run(tmp_pa "unknown": [[1, "a"]], "attention": [], "notes": [], - "deferred_runs": 0, "rate_limited_runs": 0, "job_status_failure_runs": {}, } @@ -2986,7 +3094,6 @@ def test_build_job_statuses_reports_each_finished_job(): } == { "Static checks": "success", "Postgres tests: core / A": "failure", - "Special tests": "failure", } assert statuses[f"{JOB_STATUS_CONTEXT_PREFIX}Static checks"][1] == "https://job/Static checks" @@ -3001,12 +3108,12 @@ def test_build_job_statuses_reports_the_worst_of_jobs_sharing_a_name(): assert build_job_statuses(jobs)[f"{JOB_STATUS_CONTEXT_PREFIX}Tests"][:2] == ("failure", "bad") [email protected]("name", ["Tests (AMD)", "CI passed (fork)", "x\nCI passed"]) [email protected]("name", ["Tests (AMD)", "CI passed (fork)", "x\nCI passed", "Cancelled jobs"]) def test_build_job_statuses_cannot_post_the_reconciler_contexts(name): (context,) = build_job_statuses([_fork_job(name)]) assert context.startswith(JOB_STATUS_CONTEXT_PREFIX) - assert context not in (STATUS_CONTEXT, CI_PASSED_CONTEXT) + assert context not in (STATUS_CONTEXT, CANCELLED_JOBS_CONTEXT, CI_PASSED_CONTEXT) assert "\n" not in context @@ -3030,21 +3137,21 @@ def _reporter_client(client, jobs, current=()): def test_job_status_reporter_posts_changed_statuses_failures_first(client): - already = { - "context": f"{JOB_STATUS_CONTEXT_PREFIX}Static checks", - "state": "success", - "target_url": "https://job/Static checks", - "description": "Passed in the fork - see Details", + failed_before = { + "context": f"{JOB_STATUS_CONTEXT_PREFIX}Helm tests", + "state": "failure", + "target_url": "https://job/Helm tests", + "description": "Failed in the fork - see Details", } _reporter_client( client, [_fork_job("Static checks"), _fork_job("Helm tests"), _fork_job("Unit", conclusion="failure")], - [already], + [failed_before], ) key = JobStatusReporter(client, repo=REPO).report(_pr(), COMPLETED_RUN) - assert key == "1:1:FAILURE" + assert key == "1:1:FAILURE:2:0" assert [(c.args[2], c.kwargs["context"]) for c in client.set_status.call_args_list] == [ ("failure", f"{JOB_STATUS_CONTEXT_PREFIX}Unit"), ("success", f"{JOB_STATUS_CONTEXT_PREFIX}Helm tests"), @@ -3052,7 +3159,7 @@ def test_job_status_reporter_posts_changed_statuses_failures_first(client): def test_job_status_reporter_leaves_what_does_not_fit_in_the_budget_for_the_next_run(client): - _reporter_client(client, [_fork_job("A", conclusion="failure"), _fork_job("B"), _fork_job("C")]) + _reporter_client(client, [_fork_job(name, conclusion="failure") for name in "ABC"]) reporter = JobStatusReporter(client, repo=REPO, budget=2) assert reporter.report(_pr(), COMPLETED_RUN) is None @@ -3065,7 +3172,9 @@ def test_job_status_reporter_leaves_what_does_not_fit_in_the_budget_for_the_next def test_job_status_reporter_does_not_record_a_run_in_progress(client): - _reporter_client(client, [_fork_job("A"), _fork_job("B", status="in_progress", conclusion=None)]) + _reporter_client( + client, [_fork_job("A", conclusion="failure"), _fork_job("B", status="in_progress", conclusion=None)] + ) assert JobStatusReporter(client, repo=REPO).report(_pr(), RUNNING_RUN) is None client.set_status.assert_called_once() @@ -3073,19 +3182,31 @@ def test_job_status_reporter_does_not_record_a_run_in_progress(client): @pytest.mark.parametrize( - ("marker_jobs", "marker_sha", "reads"), + ("marker_jobs", "marker_sha", "reads", "expected"), [ - pytest.param("1:1:FAILURE", "abc123", False, id="already-reported"), - pytest.param("1:1:SUCCESS", "abc123", True, id="re-run-with-another-conclusion"), - pytest.param("1:2:FAILURE", "abc123", True, id="another-attempt-with-the-same-conclusion"), - pytest.param("1:1:FAILURE", "other", True, id="marker-of-another-commit"), + pytest.param("1:1:FAILURE:7:1", "abc123", False, "1:1:FAILURE:7:1", id="already-reported"), + pytest.param("1:1:FAILURE", "abc123", True, "1:1:FAILURE:0:0", id="recorded-before-the-counts"), + pytest.param( + "1:1:FAILURE:7", "abc123", True, "1:1:FAILURE:0:0", id="recorded-with-the-passed-count-only" + ), + pytest.param( + "1:1:SUCCESS:7:1", "abc123", True, "1:1:FAILURE:0:0", id="re-run-with-another-conclusion" + ), + pytest.param( + "1:2:FAILURE:7:1", + "abc123", + True, + "1:1:FAILURE:0:0", + id="another-attempt-with-the-same-conclusion", + ), + pytest.param("1:1:FAILURE:7:1", "other", True, "1:1:FAILURE:0:0", id="marker-of-another-commit"), ], ) -def test_job_status_reporter_reads_a_completed_run_once(client, marker_jobs, marker_sha, reads): +def test_job_status_reporter_reads_a_completed_run_once(client, marker_jobs, marker_sha, reads, expected): _reporter_client(client, []) pr = _pr(marker=dataclasses.replace(_marker("failed", sha=marker_sha), jobs=marker_jobs)) - assert JobStatusReporter(client, repo=REPO).report(pr, COMPLETED_RUN) == "1:1:FAILURE" + assert JobStatusReporter(client, repo=REPO).report(pr, COMPLETED_RUN) == expected assert client.rest_get.called is reads @@ -3195,7 +3316,7 @@ def test_job_status_reporter_leaves_other_contexts_alone(client): def test_job_status_reporter_charges_each_write_even_when_a_later_one_fails(client): - _reporter_client(client, [_fork_job("A", conclusion="failure"), _fork_job("B"), _fork_job("C")]) + _reporter_client(client, [_fork_job(name, conclusion="failure") for name in "ABC"]) client.set_status.side_effect = [None, GitHubError("HTTP 500")] reporter = JobStatusReporter(client, repo=REPO, budget=10) @@ -3226,6 +3347,21 @@ def _reconcile_with_reporter(client, tmp_path, config=CONFIG, reported_file=None ) [email protected]("airflow_breeze.utils.fork_ci_reconcile.find_job_counts", autospec=True) [email protected]("airflow_breeze.utils.fork_ci_reconcile.JobStatusReporter", autospec=True) [email protected]("airflow_breeze.utils.fork_ci_reconcile.classify", autospec=True) +def test_run_reconcile_shows_the_job_counts(mock_classify, mock_reporter, mock_counts, client, tmp_path): + mock_classify.return_value = Plan(state=ForkState.RUNNING) + mock_reporter.return_value.report.return_value = None + mock_reporter.return_value.deferred = [] + mock_reporter.return_value.delayed = [] + mock_counts.return_value = JobCounts(17, 2) + + _reconcile_with_reporter(client, tmp_path) + + assert mock_classify.call_args.kwargs["job_counts"] == JobCounts(17, 2) + + @mock.patch("airflow_breeze.utils.fork_ci_reconcile.JobStatusReporter", autospec=True) @mock.patch("airflow_breeze.utils.fork_ci_reconcile.classify", autospec=True) def test_run_reconcile_classifies_when_reporting_job_statuses_fails( @@ -3316,42 +3452,239 @@ def test_job_status_reporter_does_not_read_without_writes_left(client): assert reporter.deferred == [42] [email protected](("deferred", "has_alerts"), [([1], True), ([], False)]) @mock.patch("airflow_breeze.utils.fork_ci_reconcile.JobStatusReporter", autospec=True) @mock.patch("airflow_breeze.utils.fork_ci_reconcile.classify", autospec=True) -def test_run_reconcile_reports_deferred_job_statuses_to_slack( - mock_classify, mock_reporter, client, tmp_path, deferred, has_alerts +def test_run_reconcile_does_not_alert_on_deferred_job_statuses( + mock_classify, mock_reporter, client, tmp_path ): mock_classify.return_value = Plan(state=ForkState.RUNNING) mock_reporter.return_value.report.return_value = None - mock_reporter.return_value.deferred = deferred + mock_reporter.return_value.deferred = [1] mock_reporter.return_value.delayed = [] - assert _reconcile_with_reporter(client, tmp_path) is has_alerts + assert _reconcile_with_reporter(client, tmp_path) is False - if has_alerts: - assert JOB_STATUSES_DEFERRED_NOTE in (tmp_path / "s.json").read_text() + [email protected]( + ("since_previous_run", "budget"), + [ + pytest.param(None, 20, id="no-previous-run"), + pytest.param(timedelta(minutes=5), 20, id="on-the-cron"), + pytest.param(timedelta(minutes=9, seconds=30), 38, id="started-late"), + pytest.param(timedelta(minutes=1), 4, id="queued-behind-the-previous-run"), + pytest.param(timedelta(hours=3), 40, id="after-a-gap"), + pytest.param(timedelta(minutes=-1), 0, id="clock-skew"), + ], +) +def test_compute_job_status_budget(since_previous_run, budget): + assert compute_job_status_budget(since_previous_run) == budget + + +def test_job_status_budget_after_a_gap_is_capped(): + assert compute_job_status_budget(MAX_JOB_STATUS_BUDGET_INTERVAL) == compute_job_status_budget( + MAX_JOB_STATUS_BUDGET_INTERVAL * 2 + ) + + +MONITOR_RUN_URL = "https://github.com/apache/airflow/actions/runs/100" +def _monitor_run(run_id, started, status="completed", attempt=1): + return {"id": run_id, "status": status, "run_started_at": started, "run_attempt": attempt} + + [email protected]( + ("runs", "expected"), + [ + pytest.param( + [ + _monitor_run(102, "2026-11-10T11:59:00Z", status="pending"), + _monitor_run(100, "2026-11-10T11:58:00Z", status="in_progress"), + _monitor_run(99, "2026-11-10T11:51:00Z"), + _monitor_run(98, "2026-11-10T11:42:00Z"), + ], + timedelta(minutes=7), + id="previous-started-run", + ), + pytest.param([_monitor_run(100, "2026-11-10T11:58:00Z")], None, id="first-run"), + pytest.param([_monitor_run(99, "2026-11-10T11:51:00Z")], None, id="this-run-not-listed"), + ], +) +def test_find_time_since_previous_monitor_run(client, runs, expected): + client.rest_get.return_value = {"workflow_runs": runs} + + assert find_time_since_previous_monitor_run(client, REPO, run_url=MONITOR_RUN_URL) == expected + client.rest_get.assert_called_once_with( + f"repos/{REPO}/actions/workflows/fork-ci-monitor.yml/runs?per_page=10" + ) + + [email protected]( + ("previous_attempt_start", "expected"), + [ + pytest.param("2026-11-10T12:03:00Z", timedelta(minutes=3), id="counts-from-the-previous-attempt"), + pytest.param("2026-11-10T11:50:00Z", timedelta(minutes=6), id="later-run-in-between"), + ], +) +def test_find_time_since_previous_monitor_run_for_a_re_run(client, previous_attempt_start, expected): + def rest_get(path): + if path.endswith("/actions/runs/100/attempts/2"): + return {"id": 100, "run_attempt": 2, "run_started_at": previous_attempt_start} + return { + "workflow_runs": [ + _monitor_run(100, "2026-11-10T12:06:00Z", status="in_progress", attempt=3), + _monitor_run(99, "2026-11-10T12:00:00Z"), + ] + } + + client.rest_get.side_effect = rest_get + + assert find_time_since_previous_monitor_run(client, REPO, run_url=MONITOR_RUN_URL) == expected + + +def test_find_time_since_previous_monitor_run_when_the_previous_attempt_cannot_be_read(client): + client.rest_get.side_effect = [ + {"workflow_runs": [_monitor_run(100, "2026-11-10T12:06:00Z", status="in_progress", attempt=2)]}, + GitHubError("gh api failed: HTTP 502"), + ] + + assert find_time_since_previous_monitor_run(client, REPO, run_url=MONITOR_RUN_URL) is None + + +def test_find_time_since_previous_monitor_run_without_a_run_id(client): + assert find_time_since_previous_monitor_run(client, REPO, run_url="https://run") is None + client.rest_get.assert_not_called() + + +def test_find_time_since_previous_monitor_run_when_the_runs_cannot_be_read(client): + client.rest_get.side_effect = GitHubError("gh api failed: HTTP 502") + + assert find_time_since_previous_monitor_run(client, REPO, run_url=MONITOR_RUN_URL) is None + + [email protected]("airflow_breeze.utils.fork_ci_reconcile.find_time_since_previous_monitor_run", autospec=True) @mock.patch("airflow_breeze.utils.fork_ci_reconcile.JobStatusReporter", autospec=True) [email protected]("airflow_breeze.utils.fork_ci_reconcile.report_project_ci_runs_started", autospec=True) @mock.patch("airflow_breeze.utils.fork_ci_reconcile.classify", autospec=True) -def test_run_reconcile_reports_deferred_job_statuses_only_when_deferring_persists( - mock_classify, mock_reporter, client, tmp_path +def test_run_reconcile_budgets_job_statuses_for_the_time_since_the_previous_run( + mock_classify, mock_report, mock_reporter, mock_previous, client, tmp_path ): mock_classify.return_value = Plan(state=ForkState.RUNNING) + mock_report.return_value = [] mock_reporter.return_value.report.return_value = None - client.dry_run = False - reported_file = tmp_path / "reported.json" - reported_file.write_text("{}") - alerts = [] - for deferred in ([1], [1], [], *[[1]] * DEFERRED_RUNS_BEFORE_ALERT): - mock_reporter.return_value.deferred = deferred - mock_reporter.return_value.delayed = [] - alerts.append(_reconcile_with_reporter(client, tmp_path, reported_file=reported_file)) + mock_reporter.return_value.deferred = [] + mock_reporter.return_value.delayed = [] + mock_previous.return_value = timedelta(minutes=9) + + _reconcile_with_reporter(client, tmp_path) + + assert mock_reporter.call_args.kwargs["budget"] == compute_job_status_budget(timedelta(minutes=9)) + + +def test_count_jobs(): + jobs = [ + _fork_job("A"), + _fork_job("B"), + _fork_job("C", conclusion="failure"), + _fork_job("D", conclusion="skipped"), + _fork_job("E", status="in_progress", conclusion=None), + _fork_job("F", conclusion="cancelled"), + _fork_job("G", status="in_progress", conclusion="cancelled"), + _fork_job("Platform: AMD"), + _fork_job("CI passed", conclusion="cancelled"), + ] + + assert count_jobs(jobs) == JobCounts(passed=2, cancelled=1) + + [email protected]( + ("fork", "read_jobs", "jobs_reported", "expected"), + [ + pytest.param(COMPLETED_RUN, {1: [_fork_job("A")]}, None, JobCounts(1), id="read"), + pytest.param( + COMPLETED_RUN, {1: [_fork_job("A")]}, "1:1:FAILURE:9:3", JobCounts(1), id="read-wins-over-the-key" + ), + pytest.param(COMPLETED_RUN, {}, "1:1:FAILURE:9:3", JobCounts(9, 3), id="from-the-recorded-key"), + pytest.param(COMPLETED_RUN, {}, "1:1:FAILURE", None, id="key-recorded-before-the-counts"), + pytest.param(COMPLETED_RUN, {}, "1:1:FAILURE:9:x", None, id="malformed-key"), + pytest.param(COMPLETED_RUN, {}, None, None, id="not-read"), + pytest.param(COMPLETED_RUN, {1: None}, None, None, id="read-failed"), + pytest.param(ForkObservation(), {1: [_fork_job("A")]}, "1:1:FAILURE:9:3", None, id="no-run"), + ], +) +def test_find_job_counts(fork, read_jobs, jobs_reported, expected): + assert find_job_counts(fork, read_jobs, jobs_reported) == expected + + +def test_job_status_reporter_counts_cancelled_jobs_instead_of_posting_them(client): + failed_before = { + "context": f"{JOB_STATUS_CONTEXT_PREFIX}Helm tests", + "state": "failure", + "target_url": "https://job/Helm tests", + "description": "Failed in the fork - see Details", + } + _reporter_client( + client, + [ + _fork_job("Unit", conclusion="failure"), + _fork_job("Helm tests", conclusion="cancelled"), + _fork_job("MySQL tests", conclusion="cancelled"), + ], + [failed_before], + ) + + key = JobStatusReporter(client, repo=REPO).report(_pr(), COMPLETED_RUN) + + assert key == "1:1:FAILURE:0:2" + # One status for the cancelled jobs; the earlier failure of a job the run cancelled is kept rather + # than marked as not in the run. + assert [c.args[2:] + (c.kwargs["context"],) for c in client.set_status.call_args_list] == [ + ("failure", RUN_URL, "2 jobs cancelled in the fork - see Details", CANCELLED_JOBS_CONTEXT), + ( + "failure", + "https://job/Unit", + "Failed in the fork - see Details", + f"{JOB_STATUS_CONTEXT_PREFIX}Unit", + ), + ] + + [email protected]( + ("current", "posted"), + [ + pytest.param((), [], id="nothing-cancelled-before"), + pytest.param( + [ + { + "context": CANCELLED_JOBS_CONTEXT, + "state": "failure", + "target_url": "https://github.com/someone/airflow/actions/runs/0", + "description": "3 jobs cancelled in the fork - see Details", + } + ], + [("success", RUN_URL, "No jobs cancelled in the latest fork run")], + id="cleared-after-a-re-run", + ), + pytest.param( + [ + { + "context": CANCELLED_JOBS_CONTEXT, + "state": "success", + "target_url": "https://github.com/someone/airflow/actions/runs/0", + "description": "No jobs cancelled in the latest fork run", + } + ], + [], + id="already-cleared", + ), + ], +) +def test_job_status_reporter_posts_the_cancelled_jobs_status_only_when_needed(client, current, posted): + _reporter_client(client, [_fork_job("A")], current) + + JobStatusReporter(client, repo=REPO).report(_pr(), COMPLETED_RUN) - assert alerts == [*[False] * (DEFERRED_RUNS_BEFORE_ALERT + 2), True] - assert JOB_STATUSES_DEFERRED_NOTE in (tmp_path / "s.json").read_text() - assert json.loads(reported_file.read_text())["deferred_runs"] == DEFERRED_RUNS_BEFORE_ALERT + assert [c.args[2:] for c in client.set_status.call_args_list] == posted def test_build_job_statuses_treats_contexts_case_insensitively(): @@ -3373,7 +3706,7 @@ def test_job_status_reporter_compares_contexts_case_insensitively(client): } _reporter_client(client, [_fork_job("Tests", conclusion="failure")], [posted]) - assert JobStatusReporter(client, repo=REPO).report(_pr(), COMPLETED_RUN) == "1:1:FAILURE" + assert JobStatusReporter(client, repo=REPO).report(_pr(), COMPLETED_RUN) == "1:1:FAILURE:0:0" client.set_status.assert_not_called() @@ -3998,8 +4331,8 @@ def test_run_reconcile_retries_a_pull_request_whose_head_moved(client, tmp_path) @pytest.mark.parametrize( ("fails", "job_status_budget"), [ - pytest.param(False, MAX_JOB_STATUSES_PER_RUN - 2, id="reported"), - pytest.param(True, MAX_JOB_STATUSES_PER_RUN - MAX_CI_RUN_STARTED_PER_RUN, id="fails"), + pytest.param(False, compute_job_status_budget(None) - 2, id="reported"), + pytest.param(True, compute_job_status_budget(None) - MAX_CI_RUN_STARTED_PER_RUN, id="fails"), ], ) @mock.patch("airflow_breeze.utils.fork_ci_reconcile.JobStatusReporter", autospec=True)
