This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v3-4-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/v3-4-test by this push:
new 727ac291743 [v3-4-test] Count passed fork CI jobs and post job
statuses at the intended rate (#77125) (#77140)
727ac291743 is described below
commit 727ac291743d1b8014b9ccb3c70c52ec12a18eab
Author: Jarek Potiuk <[email protected]>
AuthorDate: Mon Nov 16 01:29:44 2026 +0100
[v3-4-test] Count passed fork CI jobs and post job statuses at the intended
rate (#77125) (#77140)
* Post fork CI job statuses at the intended hourly rate
GitHub starts the 5-minute monitor only every 9-10 minutes, so a fixed
13 writes a run posted about half the job statuses an hour the budget
was sized for, and a few full-matrix fork runs at once took hours to
show their jobs. A run that only deferred them also alerted Slack after
nine runs, although each run made progress; a run whose statuses are
still missing an hour after it finished is already reported on its own.
Generated-by: Claude Opus 5
Claude-Session: https://claude.ai/code/session_01NyQq9U7i5HhjDgrQ9qd3gq
* Count passed fork jobs instead of posting a status for each
A full-matrix fork run has about 130 jobs, and posting each passed one
as its own commit status spent most of the shared write budget on what
the aggregate status already says. Failures still get their own status,
which is what a reviewer looks for.
The time since the previous monitor run is read from GitHub, because
the record carried between runs stays unchanged after a failed run or
Slack post, which would have let the next run spend that budget again.
Generated-by: Claude Opus 5
Claude-Session: https://claude.ai/code/session_01NyQq9U7i5HhjDgrQ9qd3gq
* Measure the job status budget between monitor run starts
Counting from when reconciling starts credited each run's setup time
again, and a re-run credited the time its first attempt already had,
so the hourly budget could be exceeded.
Generated-by: Claude Opus 5
Claude-Session: https://claude.ai/code/session_01NyQq9U7i5HhjDgrQ9qd3gq
* Count a monitor re-run from its previous attempt
Counting from when the run was created credited each later attempt
with the time earlier attempts already had.
Generated-by: Claude Opus 5
Claude-Session: https://claude.ai/code/session_01NyQq9U7i5HhjDgrQ9qd3gq
* Keep the passed-jobs count when the fork status write fails
The state comment records a completed run before the status is posted,
and later runs do not read that run's jobs again, so a failed status
write would have lost the count for good.
Generated-by: Claude Opus 5
Claude-Session: https://claude.ai/code/session_01NyQq9U7i5HhjDgrQ9qd3gq
* Count cancelled fork jobs instead of posting a status for each
A fork run cancelled by a newer push or a failing job cancels dozens of
jobs at once, and each one became a red status that said no more than
a count does, while spending the shared write budget.
Generated-by: Claude Opus 5
Claude-Session: https://claude.ai/code/session_01NyQq9U7i5HhjDgrQ9qd3gq
* Report a fork run's cancelled jobs in one status
The count in Fork CI / Tests (AMD) is easy to miss, while a status of
its own shows in the pull request's checks that jobs were cancelled.
A marker holding only the passed count now has its jobs read again,
since it cannot give the cancelled count.
Generated-by: Claude Opus 5
Claude-Session: https://claude.ai/code/session_01NyQq9U7i5HhjDgrQ9qd3gq
(cherry picked from commit 748148ccdb7f2383528627d9972f8f5f06a85d29)
---
dev/AGENTS.md | 12 +-
.../doc/ci/08_fork_ci_for_external_contributors.md | 105 +++--
.../src/airflow_breeze/utils/fork_ci_messages.py | 50 ++-
.../src/airflow_breeze/utils/fork_ci_reconcile.py | 220 +++++++++--
dev/breeze/tests/test_fork_ci_messages.py | 66 ++++
dev/breeze/tests/test_fork_ci_reconcile.py | 427 ++++++++++++++++++---
6 files changed, 746 insertions(+), 134 deletions(-)
diff --git a/dev/AGENTS.md b/dev/AGENTS.md
index c2ac0cbbb2f..66e0218a5ca 100644
--- a/dev/AGENTS.md
+++ b/dev/AGENTS.md
@@ -47,17 +47,17 @@ Document `uv run` (not `python`) as the invocation method
in READMEs and instruc
## Fork CI job statuses and GitHub's rate limits
The fork CI monitor (`breeze ci fork-ci-reconcile`, run by
`.github/workflows/fork-ci-monitor.yml`)
-reports every job of a gated pull request's fork run as its own commit status.
Those writes are the
+reports every failed job of a gated pull request's fork run as its own commit
status, counts the
+passed and cancelled jobs in `Fork CI / Tests (AMD)`, and reports the
cancelled ones in one
+`Fork CI / Cancelled jobs` status. Those writes are the
largest share of what the monitor does with the repository's `GITHUB_TOKEN`,
whose rate limits every
workflow of `apache/airflow` shares (see "Write budget" in
`dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md`).
If the budget is exceeded, turn the job statuses off: set `job_statuses =
false` in
-`.github/fork-ci-gate.toml`. Signs that it is exceeded are the monitor's Slack
note that job
-statuses "have not fit in the budget" (posted only after job statuses were
deferred for about 45 minutes in a row)
-staying for hours, pull requests listed because their fork run finished more
than an hour ago with
-job statuses still not posted, its Slack note that GitHub rate-limited it for
about 30 minutes in a row, the
+`.github/fork-ci-gate.toml`. Signs that it is exceeded are pull requests
listed because their fork
+run finished more than an hour ago with job statuses still not posted, staying
for hours, its Slack note that GitHub rate-limited it for about 30 minutes in a
row, the
"Rate limits left" line of its run log approaching zero, or other workflows of
the repository
failing on secondary rate limits. The aggregate `Fork CI / Tests (AMD)`
status and `CI passed` are posted either way, so turning job statuses off
loses detail, not gating.
-Lowering `MAX_JOB_STATUSES_PER_RUN` in `fork_ci_reconcile.py` is the softer
alternative.
+Lowering `JOB_STATUSES_PER_HOUR` in `fork_ci_reconcile.py` is the softer
alternative.
diff --git a/dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md
b/dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md
index a1d932971a5..a9c7f42a1d2 100644
--- a/dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md
+++ b/dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md
@@ -886,7 +886,15 @@ and `gate_bypassed` (`error`) and `failed` (`failure`),
yellow for `awaiting_pus
when one exists and the fork's Actions page for `ci-amd.yml` otherwise, so a
maintainer can
open the fork run in one click; for `gate_bypassed` it is the project CI run
that was
observed. The `description` is a short sentence under 140 characters that
always ends with
-where to look next.
+where to look next. For `running`, `failed`, `green` and `green_unverified` it
is followed by the
+fork run's passed and cancelled jobs, such as "Fork CI passed - see Details
(127 jobs passed)" or
+"Fork CI failed - see Details (97 jobs passed, 31 cancelled)", since neither
is posted as statuses
+of their own (see "Job statuses"). The counts are taken from the jobs the run
read for its job
+statuses, or else from the marker of a completed run whose jobs are all
reported, so a status
+write that failed after the state comment recorded the run still gets them;
otherwise the status
+keeps the counts it shows, and they are never carried over from another state.
The reconciler
+reads the state back from the description without the counts, and accepts
counts only on those
+four states.
Statuses are immutable and per SHA. The reconciler compares the desired
`(state,
target_url, description)` with the current one fetched in the pass 2 query and
posts only on
@@ -900,8 +908,10 @@ monitor run rather than `details_url`. A status honours
`target_url`.
### Job statuses
-Next to the aggregate status, each job of the fork run is reported as its own
commit status, so
-the pull request shows which jobs run, failed or passed, each linking to the
job in the fork.
+Next to the aggregate status, each failed job of the fork run is reported as
its own commit
+status, so the pull request shows which jobs failed, each linking to the job
in the fork. Passed
+jobs are counted in `Fork CI / Tests (AMD)` instead, and cancelled jobs in it
and in one status of
+their own, `Fork CI / Cancelled jobs`.
The run is the one `Fork CI / Tests (AMD)` reports (its `databaseId` is
fetched with the check
suites); its jobs come from `GET
/repos/<fork>/actions/runs/<id>/jobs?filter=latest`.
@@ -910,9 +920,20 @@ suites); its jobs come from `GET
/repos/<fork>/actions/runs/<id>/jobs?filter=lat
the prefix is what keeps a job from posting `Fork CI / Tests (AMD)`, from
which the reconciler
reads its decisions back, or `CI passed`, which the merge ruleset requires.
Jobs that share a
context, which GitHub compares case-insensitively, are reported as the worst
of them.
-- Only finished jobs are reported: a successful one as `success`, and a
failed, timed-out,
- cancelled or unstartable one as `failure`. Queued and running jobs are not,
so each job costs one
- write instead of two; `Fork CI / Tests (AMD)` shows the run as running,
linking it, and with a
+- Only finished jobs are reported: a failed, timed-out or unstartable one as
`failure`, and a
+ successful one as `success` only over a status of the same job that is not a
success, such as
+ its failure in an earlier run of the commit, which would otherwise keep
showing. A full-matrix
+ run has about 130 jobs and each status is a write, so posting every passed
job would take most
+ of the budget for what the aggregate status already says; on 2026-11-15 four
such runs at once
+ took about six hours to show.
+- Cancelled jobs get no status each: a run cancelled by a newer push or by a
failing job cancels
+ dozens at once, and each would be a red status saying no more than a count
does. Instead
+ `Fork CI / Cancelled jobs`, linking the fork run, is `failure` with "31 jobs
cancelled in the
+ fork - see Details" while the run has cancelled jobs, and is overwritten
with `success` and "No
+ jobs cancelled in the latest fork run" once a later run of the commit has
none; it is not posted
+ for a run that never had any. The context is outside the job statuses' `Fork
CI job /` prefix, so no job can
+ post it. A job the run cancelled keeps what an earlier run of the commit
reported for it. Queued and running jobs are not
+ reported, so each job costs at most one write; `Fork CI / Tests (AMD)` shows
the run as running, linking it, and with a
5-minute cadence a per-job "running" status was behind the fork's own run
page anyway. A job of
a re-run in the fork keeps the result of the earlier attempt until it
finishes. Skipped and
neutral jobs, and the jobs every run has (`Fork CI gate`, `Check if the run
is needed`,
@@ -921,33 +942,31 @@ suites); its jobs come from `GET
/repos/<fork>/actions/runs/<id>/jobs?filter=lat
(`GET /repos/apache/airflow/commits/<sha>/status`), and only differences are
posted, failures
first. GraphQL has no mutation for commit statuses (`createCheckRun` exists,
but see above for
why check runs do not fit), so each one is a REST call, and they count
towards GitHub's limits on
- content-creating requests: a reconciler run posts at most
`MAX_JOB_STATUSES_PER_RUN` (13) of
- them, each write charged as it is made. What does not fit is posted by the
next run, and once
- the budget is spent no further pull request's jobs or statuses are read in
that run. One fork run
- of the full matrix (about 130 jobs) needs more than a run's budget on its
own, which the next
- run catches up on, so deferring alone is not a problem: the run log lists
the deferred pull
- requests, and only when nine runs in a row deferred, about 45 minutes of the
budget not
- keeping up, is a note added to the Slack message. The count is kept in the
record the monitor
- carries between runs (`--reported-file`); without a readable record the note
is added at once.
- Its wording is fixed, so it is posted once while deferrals persist. A pull
request whose fork run
- finished more than an hour ago (the `updatedAt` of its completed check
suite) and still has
- statuses waiting is listed on its own as an attention item, so a delay that
a busy backlog keeps
- pushing back shows which pull request it holds up; the item names the run,
so it is posted once
- for that run.
+ content-creating requests: a reconciler run posts at most its share of
`JOB_STATUSES_PER_HOUR`
+ (240) for the time since the previous run, each write charged as it is made
(see "Write
+ budget"). What does not fit is posted by a later run, and once the budget is
spent no further
+ pull request's jobs or statuses are read in that run. Deferring alone is not
reported, since
+ later runs catch up and many failing runs at once can keep the monitor
deferring while every
+ run makes progress: the run log lists the deferred pull requests. A pull
request whose fork run finished more than an hour
+ ago (the `updatedAt` of its completed check suite) and still has statuses
waiting is listed on
+ its own as an attention item, so a delay that a busy backlog keeps pushing
back shows which pull
+ request it holds up; the item names the run, so it is posted once for that
run.
- Commit statuses cannot be deleted and have no neutral state. When a run has
completed, a job
context posted for an earlier run of the same commit that the completed run
does not have, or
skipped, is overwritten with `success` and the description "Not in the
latest fork run - see
- Fork CI / Tests (AMD)", so it no longer shows as running or failed.
+ Fork CI / Tests (AMD)", so it no longer shows as running or failed; one that
is already a
+ success is left as it is.
- Once every job of a completed run is reported, the run is recorded in the
marker as
- `jobs: "<run id>:<attempt>:<conclusion>"` (`runAttempt` is fetched with the
check suites), so
- later runs do not read its jobs again. A run in progress is read on every
run, and a re-run in
+ `jobs: "<run id>:<attempt>:<conclusion>:<passed jobs>:<cancelled jobs>"`
(`runAttempt` is
+ fetched with the check suites), so later runs do not read its jobs again. A
marker recorded
+ before the counts were added has none, and its run's jobs are read once
more. A run in progress is read on every run, and a re-run in
the fork is read again because its attempt changes, even when it ends with
the same conclusion.
- Reporting job statuses is caught on its own: when it fails, the pull request
is still
classified, so for example a `CI passed` that no longer holds is still
revoked. Reading a fork
run's jobs fails now and then on GitHub server errors (HTTP 502, a reset
HTTP/2 stream) that the
next run gets past, so each failure is only logged, and the pull request is
listed with the
unknown items for Slack once its job statuses failed in nine runs in a row,
about 45 minutes. The per pull request
- count is kept in the same record as that of deferred job statuses; without a
readable record it
+ count is kept in the record the monitor carries between runs
(`--reported-file`); without a readable record it
starts again. The item's wording is fixed whatever the error, so it is
posted once while the
failure persists, and the run log has each run's error.
@@ -962,7 +981,7 @@ records this for agents.
Each run logs what is left of the token's REST and GraphQL quotas (`GET
/rate_limit`, which does
not count against them) and how many of its calls GitHub rate-limited; the
client retries those with
backoff. Rate limiting in six runs in a row, about 30 minutes, adds a note to
the Slack message, with fixed wording so
-it is posted once while it persists; the count is kept in the same record as
that of deferred job
+it is posted once while it persists; the count is kept in the same record as
that of failing job
statuses.
#### Write budget
@@ -979,7 +998,10 @@ jobs whose conclusion is not `skipped`, minus the four
jobs that are not reporte
| CI change (full tests) |
[40207881565](https://github.com/apache/airflow/actions/runs/40207881565) | 131
| 127 |
`apache/airflow` had 1898 `Tests (AMD)` pull request runs from 2026-10-18 to
2026-10-24, about 270 a
-day or 11 an hour. Each reported job costs one write, its result.
+day or 11 an hour. Each reported job costs one write, its result. Since passed
and cancelled jobs
+are only counted in `Fork CI / Tests (AMD)`, a push costs one write per failed
job (plus one per
+job that passes after failing), and the estimates below, which assume every
job is posted, are an
+upper bound.
The limits that matter are GitHub's documented ones for the `GITHUB_TOKEN` of
`apache/airflow`,
shared by every workflow of the repository: 1000 REST requests an hour, and 80
content-creating
@@ -990,14 +1012,29 @@ requests a minute and 500 an hour. Assuming about 35
reported jobs for an averag
monitor runs.
- **Gate enabled**, assuming about half of the pull request runs come from
gated pull requests
(about 5-6 pushes an hour): about 200 writes an hour without a cap, 40% of
the content-creating
- limit and more at peaks. The cap of 13 writes a run, twelve runs an hour,
holds job statuses to at
- most 156 an hour: 31% of the content-creating limit, leaving the rest to the
reconciler's other
- writes (labels, comments, the aggregate statuses) and to the other workflows
that share it. Each
- run posts at most 13 statuses at about one a second, well under the
80-a-minute limit. Job
- statuses then lag behind at busy times, the aggregate status does not. The
`CI run started`
- statuses of pull requests whose CI runs in `apache/airflow` come out of the
same 13 writes a run,
- first and at most 6 of them (see "The `CI run started` check"), so they do
not raise the total;
- at about 11 pull request runs an hour they take about 1 write a run.
+ limit and more at peaks. `JOB_STATUSES_PER_HOUR` holds job statuses to at
most 240 an hour: 48%
+ of the content-creating limit, leaving the rest to the reconciler's other
writes (labels,
+ comments, the aggregate statuses) and to the other workflows that share it.
Job statuses then
+ lag behind at busy times, the aggregate status does not. The `CI run
started` statuses of pull
+ requests whose CI runs in `apache/airflow` come out of the same budget,
first and at most 6 a
+ run (see "The `CI run started` check"), so they do not raise the total; at
about 11 pull
+ request runs an hour they take about 1 write a run.
+
+A run gets the hourly budget for the time between the previous monitor run's
start and its own,
+not one twelfth of it. Both are read from the monitor's runs (`GET
+/repos/apache/airflow/actions/workflows/fork-ci-monitor.yml/runs`, runs still
waiting in its
+concurrency group left out), so they are the same boundary of each run, setup
included, and not
+from the record the monitor carries between runs, which a failed run or a
failed Slack post leaves
+unchanged although that run may have posted statuses. A re-run counts from its
previous
+attempt's start (`GET /repos/apache/airflow/actions/runs/<id>/attempts/<n>`),
so the time earlier
+attempts had is not credited again. GitHub
+starts scheduled workflows later than their cron: on 2026-11-15 the 5-minute
monitor ran every 9
+to 10 minutes, so a fixed 13 writes a run (160 an hour at the cron's cadence)
posted only about 82
+an hour, and four full-matrix fork runs at once took about six hours to catch
up. Runs queued
+behind a slow one get little, since the time between them is short. At most
+`MAX_JOB_STATUS_BUDGET_INTERVAL` (10 minutes) is counted, so a run after a
gap, such as a
+disabled workflow, posts at most 40 statuses at about one a second: half of
the 80-a-minute
+limit. When the previous run cannot be read, a run gets one cron interval's
share, 20 writes.
The per-run caps and the run counts before a Slack note are derived from
`MONITOR_INTERVAL` in
`fork_ci_reconcile.py`, which must match the monitor's cron (a test checks
it), so changing the
@@ -1238,7 +1275,7 @@ files, a jobs call and a compare call only per long
`ci-amd.yml` run of a gated
one changed-files call and one or two compare calls per head commit that turns
green, plus one
mutation per actual state change. Job statuses add a jobs call and a status
call per pull request with a fork run in
progress or a completed run not yet reported, and up to two status mutations
per job, at most
-`MAX_JOB_STATUSES_PER_RUN` per run; see "Write budget". Pull requests that
pass 1 labelled and
+the run's share of `JOB_STATUSES_PER_HOUR`; see "Write budget". Pull requests
that pass 1 labelled and
the search does not list yet add one GraphQL call per 25. Well inside
the `GITHUB_TOKEN` limits at a 5-minute cadence. The reconciler logs its call
counts so the
budget stays observable.
diff --git a/dev/breeze/src/airflow_breeze/utils/fork_ci_messages.py
b/dev/breeze/src/airflow_breeze/utils/fork_ci_messages.py
index 7378ad7f368..6ced6cf2600 100644
--- a/dev/breeze/src/airflow_breeze/utils/fork_ci_messages.py
+++ b/dev/breeze/src/airflow_breeze/utils/fork_ci_messages.py
@@ -37,9 +37,12 @@ from typing import NamedTuple
STATUS_CONTEXT = "Fork CI / Tests (AMD)"
# Each job of the fork run is reported under this prefix. Job names come from
the contributor's
-# workflow, so the prefix keeps them from posting STATUS_CONTEXT,
CI_PASSED_CONTEXT or
-# CI_RUN_STARTED_CONTEXT.
+# workflow, so the prefix keeps them from posting STATUS_CONTEXT,
CANCELLED_JOBS_CONTEXT,
+# CI_PASSED_CONTEXT or CI_RUN_STARTED_CONTEXT.
JOB_STATUS_CONTEXT_PREFIX = "Fork CI job / "
+# The fork run's cancelled jobs, as one status rather than one each: a run
cancelled by a newer push
+# or a failing job cancels dozens of jobs at once.
+CANCELLED_JOBS_CONTEXT = "Fork CI / Cancelled jobs"
JOB_NAME_MAX_LENGTH = 100
# The check the "Require CI on main" ruleset requires before a pull request
can (auto-)merge. Project
# CI's gate job reports it; for a gated pull request, which never runs project
CI, the reconciler
@@ -120,7 +123,8 @@ class Marker:
since: datetime
# Set when the head commit's fork CI turned green and its base could be
compared.
base: BaseCheck | None = None
- # The completed fork run (``<run id>:<attempt>:<conclusion>``) whose jobs
are all reported.
+ # The completed fork run (``<run id>:<attempt>:<conclusion>:<passed
jobs>``) whose jobs are all
+ # reported.
jobs: str | None = None
# The fork run (``<run id>:<attempt>``) whose failed jobs the comment
lists.
failed_run: str | None = None
@@ -539,6 +543,16 @@ JOB_STATUSES = {
STALE_JOB_STATUS = ("success", "Not in the latest fork run - see Fork CI /
Tests (AMD)")
+def cancelled_jobs_status(cancelled: int) -> tuple[str, str]:
+ """The ``(state, description)`` of the CANCELLED_JOBS_CONTEXT status for a
fork run."""
+ if cancelled:
+ return (
+ "failure",
+ f"{cancelled} job{'' if cancelled == 1 else 's'} cancelled in the
fork - see Details",
+ )
+ return "success", "No jobs cancelled in the latest fork run"
+
+
def job_status_context(job_name: str) -> str:
# A line break or control character would make an unreadable check name.
name = "".join(char if char.isprintable() else " " for char in
job_name).strip()
@@ -562,5 +576,31 @@ STATUS_STATES = {
}
-def status_description(state: str) -> str:
- return _STATUS_DESCRIPTIONS[state]
+class JobCounts(NamedTuple):
+ """The passed and cancelled jobs of a fork run, which are not posted as
job statuses of their own."""
+
+ passed: int
+ cancelled: int = 0
+
+
+# States whose `Fork CI / Tests (AMD)` description counts the fork run's
passed and cancelled jobs.
+STATES_WITH_JOB_COUNTS = frozenset({"running", "failed", "green",
"green_unverified"})
+_JOB_COUNTS_SUFFIX = re.compile(r" \((\d+) jobs? passed(?:, (\d+)
cancelled)?\)\Z")
+
+
+def status_description(state: str, job_counts: JobCounts | None = None) -> str:
+ description = _STATUS_DESCRIPTIONS[state]
+ if job_counts is None or state not in STATES_WITH_JOB_COUNTS:
+ return description
+ counts = f"{job_counts.passed} job{'' if job_counts.passed == 1 else 's'}
passed"
+ if job_counts.cancelled:
+ counts += f", {job_counts.cancelled} cancelled"
+ return f"{description} ({counts})"
+
+
+def split_job_counts(description: str) -> tuple[str, JobCounts | None]:
+ """A status description without its job counts, and the counts; None
without them."""
+ match = _JOB_COUNTS_SUFFIX.search(description)
+ if match is None:
+ return description, None
+ return description[: match.start()], JobCounts(int(match[1]), int(match[2]
or 0))
diff --git a/dev/breeze/src/airflow_breeze/utils/fork_ci_reconcile.py
b/dev/breeze/src/airflow_breeze/utils/fork_ci_reconcile.py
index 61a30366219..65b6fd75170 100644
--- a/dev/breeze/src/airflow_breeze/utils/fork_ci_reconcile.py
+++ b/dev/breeze/src/airflow_breeze/utils/fork_ci_reconcile.py
@@ -58,6 +58,7 @@ from airflow_breeze.utils.fork_ci_gate import (
)
from airflow_breeze.utils.fork_ci_github import GitHubClient, GitHubError
from airflow_breeze.utils.fork_ci_messages import (
+ CANCELLED_JOBS_CONTEXT,
CI_PASSED_CONTEXT,
CI_PASSED_PROJECT_DESCRIPTION,
CI_PASSED_REVOKED_DESCRIPTION,
@@ -68,11 +69,14 @@ from airflow_breeze.utils.fork_ci_messages import (
JOB_STATUS_CONTEXT_PREFIX,
JOB_STATUSES,
STALE_JOB_STATUS,
+ STATES_WITH_JOB_COUNTS,
STATUS_CONTEXT,
STATUS_STATES,
BaseCheck,
FailedJob,
+ JobCounts,
Marker,
+ cancelled_jobs_status,
ci_passed_description,
fork_actions_url,
gate_modified_close_comment,
@@ -83,6 +87,7 @@ from airflow_breeze.utils.fork_ci_messages import (
parse_marker,
rebase_needed_description,
released_comment,
+ split_job_counts,
state_comment,
status_description,
strip_failed_job_list,
@@ -199,9 +204,29 @@ UNREPORTED_JOB_NAMES = GATED_RUN_JOB_NAMES |
{GATE_JOB_NAME, "Platform: AMD"}
RUNNING_JOB_STATUSES = frozenset({"queued", "waiting", "pending", "requested",
"in_progress"})
JOB_STATE_RANKS = {"failure": 0, "pending": 1, "success": 2}
# Commit statuses count towards GitHub's limit of 500 content-creating
requests per hour, shared by
-# every workflow of the repository. This keeps job statuses to about a third
of it; the rest follow
-# on the next run, failures first. See "Job statuses" in the design doc.
-MAX_JOB_STATUSES_PER_RUN = 160 // RUNS_PER_HOUR
+# every workflow of the repository. This keeps job statuses under half of it;
the rest follow on a
+# later run, failures first. See "Write budget" in the design doc.
+JOB_STATUSES_PER_HOUR = 240
+# GitHub starts scheduled runs later than their cron, the monitor often only
every 9-10 minutes, so a
+# run gets the hourly budget for the time since the previous run started
rather than for
+# MONITOR_INTERVAL. At
+# most this long is counted, so a run after a gap (a disabled workflow, a lost
record) stays at half
+# of GitHub's limit of 80 content-creating requests a minute.
+MAX_JOB_STATUS_BUDGET_INTERVAL = timedelta(minutes=10)
+MONITOR_WORKFLOW_FILE = "fork-ci-monitor.yml"
+# Enough to find the previous started run behind the runs queued in the
monitor's concurrency group.
+MONITOR_RUNS_READ = 10
+NOT_STARTED_RUN_STATUSES = frozenset({"queued", "waiting", "pending",
"requested"})
+
+
+def compute_job_status_budget(since_previous_run: timedelta | None) -> int:
+ """The job status writes a run may make; without a previous run, those of
one ``MONITOR_INTERVAL``."""
+ if since_previous_run is None:
+ since_previous_run = MONITOR_INTERVAL
+ counted = min(max(since_previous_run, timedelta(0)),
MAX_JOB_STATUS_BUDGET_INTERVAL)
+ return JOB_STATUSES_PER_HOUR * counted // timedelta(hours=1)
+
+
# `CI run started` statuses of pull requests whose CI runs in apache/airflow
come out of that budget
# first: one write per new head commit, which merging waits for, while job
statuses are informational.
# At most this many, so job statuses keep the rest; the first runs after the
check became required
@@ -212,10 +237,6 @@ MAX_CI_RUN_STARTED_PER_RUN = 80 // RUNS_PER_HOUR
MAX_PROJECT_CI_PASSED_PER_RUN = max(20 // RUNS_PER_HOUR, 1)
# The GitHub Actions app, whose check suites are the workflow runs of a commit.
GITHUB_ACTIONS_APP_ID = 15368
-# A single large fork run can need more writes than one run's budget, which
the next run catches up
-# on, so deferring is only reported when it lasts this many consecutive runs:
then the budget does
-# not keep up with the pull requests.
-DEFERRED_RUNS_BEFORE_ALERT = count_runs_in(timedelta(minutes=45))
# GitHub answering with rate limits in this many consecutive runs is reported:
a single burst is
# retried, while a persisting one means the repository's shared limits are
close to exhausted.
RATE_LIMITED_RUNS_BEFORE_ALERT = count_runs_in(timedelta(minutes=30))
@@ -230,20 +251,14 @@ JOB_STATUSES_FAILED_REASON = (
"the run log has the errors"
)
# A pull request whose fork run finished this long ago with job statuses still
not all posted is
-# reported by itself: deferring is expected to catch up within a few runs, not
to leave one behind.
+# reported. Deferring alone is not: several full-matrix fork runs at once
(about 130 statuses each)
+# keep later runs deferring for hours while every one of them makes progress.
MIRROR_DELAY_BEFORE_ALERT = timedelta(hours=1)
RATE_LIMITED_NOTE = (
f"GitHub rate-limited the fork CI monitor in
{RATE_LIMITED_RUNS_BEFORE_ALERT} runs in a row; the "
"repository's shared GITHUB_TOKEN limits are close to exhausted. Consider
`job_statuses = false` in "
".github/fork-ci-gate.toml (see dev/AGENTS.md)"
)
-# Fixed text, so the note is posted to Slack once while deferrals persist; the
run log lists the pull
-# requests.
-JOB_STATUSES_DEFERRED_NOTE = (
- f"job statuses have not fit in the budget of {MAX_JOB_STATUSES_PER_RUN}
writes per run for "
- f"{DEFERRED_RUNS_BEFORE_ALERT} runs in a row and are posted by later runs;
if this persists, set "
- "`job_statuses = false` in .github/fork-ci-gate.toml (see dev/AGENTS.md)"
-)
RUNNING_STATUSES = frozenset({"QUEUED", "IN_PROGRESS", "PENDING", "WAITING",
"REQUESTED"})
FAILED_CONCLUSIONS = frozenset({"FAILURE", "CANCELLED", "TIMED_OUT",
"STARTUP_FAILURE", "ACTION_REQUIRED"})
@@ -601,20 +616,31 @@ def status_state(status: tuple[str, str, str] | None) ->
str | None:
"""The state the reconciler recorded in a commit status it posted, if it
posted it."""
if status is None:
return None
+ description, job_counts = split_job_counts(status[2])
return next(
(
state
for state, github_state in STATUS_STATES.items()
- if (github_state, status_description(state)) == (status[0],
status[2])
+ if (github_state, status_description(state)) == (status[0],
description)
+ and (job_counts is None or state in STATES_WITH_JOB_COUNTS)
),
None,
)
def _set_status(
- plan: Plan, pr: PullRequestSnapshot, state: str, target_url: str, *,
force: bool = False
+ plan: Plan,
+ pr: PullRequestSnapshot,
+ state: str,
+ target_url: str,
+ *,
+ force: bool = False,
+ job_counts: JobCounts | None = None,
) -> None:
- desired = (STATUS_STATES[state], target_url, status_description(state))
+ """:param job_counts: the fork run's job counts; None keeps the counts the
status already shows."""
+ if job_counts is None and pr.current_status is not None and
status_state(pr.current_status) == state:
+ job_counts = split_job_counts(pr.current_status[2])[1]
+ desired = (STATUS_STATES[state], target_url, status_description(state,
job_counts))
if force or desired != pr.current_status:
plan.status = desired
@@ -666,6 +692,7 @@ def classify(
find_gate_change: Callable[[], str | None] = lambda: None,
find_failed_jobs: Callable[[], list[FailedJob] | None] = lambda: [],
verify_head_repo_unavailable: Callable[[], bool] = lambda: False,
+ job_counts: JobCounts | None = None,
) -> Plan:
"""
Decide the mutations for one gated pull request.
@@ -691,6 +718,8 @@ def classify(
:param find_failed_jobs: the jobs of the fork run that did not pass, None
when they cannot be
read; only called for a failed run that started. On None the state
comment is left for the
next run, so that a failed read does not drop the list from it.
+ :param job_counts: the passed and cancelled jobs of the fork run, shown in
`Fork CI / Tests
+ (AMD)`; None when they are not known, which keeps the counts the
status shows.
"""
plan = Plan()
labels = set(pr.labels)
@@ -907,7 +936,12 @@ def classify(
target_url = fork.run_url or fork_actions_url(pr.head_repo)
startup_failure = state == ForkState.FAILED and fork.suite_conclusion ==
"STARTUP_FAILURE"
_set_status(
- plan, pr, "startup_failure" if startup_failure else state.value,
target_url, force=refresh_status
+ plan,
+ pr,
+ "startup_failure" if startup_failure else state.value,
+ target_url,
+ force=refresh_status,
+ job_counts=job_counts,
)
_set_ci_passed(plan, pr, state, base, target_url=target_url)
_set_ci_run_started(plan, pr, fork)
@@ -1849,6 +1883,39 @@ def fetch_fork_observations(
return observations
+def count_jobs(jobs: Iterable[dict[str, Any]]) -> JobCounts:
+ """The finished jobs of a fork run that passed and that were cancelled, of
those job statuses cover."""
+ conclusions = [
+ job.get("conclusion")
+ for job in jobs
+ if job["name"] not in UNREPORTED_JOB_NAMES and job.get("status") not
in RUNNING_JOB_STATUSES
+ ]
+ return JobCounts(conclusions.count("success"),
conclusions.count("cancelled"))
+
+
+def find_job_counts(
+ fork: ForkObservation, read_jobs: ReadJobs, jobs_reported: str | None
+) -> JobCounts | None:
+ """
+ The job counts of the fork run, from its jobs when this reconciler run
read them, or else from
+ the key of the completed run whose jobs are all reported.
+ """
+ if fork.run_id is None:
+ return None
+ jobs = read_jobs.get(fork.run_id)
+ if jobs is not None:
+ return count_jobs(jobs)
+ return parse_job_counts(jobs_reported)
+
+
+def parse_job_counts(jobs_reported: str | None) -> JobCounts | None:
+ """The job counts in a completed run's key, ``<run
id>:<attempt>:<conclusion>:<passed>:<cancelled>``."""
+ parts = (jobs_reported or "").split(":")
+ if len(parts) != 5 or not (parts[3].isdigit() and parts[4].isdigit()):
+ return None
+ return JobCounts(int(parts[3]), int(parts[4]))
+
+
def build_job_statuses(jobs: Iterable[dict[str, Any]]) -> dict[str, tuple[str,
str, str]]:
"""The ``(state, target URL, description)`` to report for each finished
fork job, by context."""
statuses: dict[str, tuple[str, str, str]] = {}
@@ -1858,9 +1925,10 @@ def build_job_statuses(jobs: Iterable[dict[str, Any]])
-> dict[str, tuple[str, s
if job["name"] in UNREPORTED_JOB_NAMES:
continue
outcome = "" if job.get("status") in RUNNING_JOB_STATUSES else
job.get("conclusion") or ""
- if outcome not in JOB_STATUSES:
- # Unfinished jobs are left to `Fork CI / Tests (AMD)`; skipped and
neutral ones did no work
- # worth a check on the pull request.
+ if outcome not in JOB_STATUSES or outcome == "cancelled":
+ # Unfinished and cancelled jobs are left to `Fork CI / Tests
(AMD)`, which counts the
+ # cancelled ones: a cancelled run cancels dozens at once. Skipped
and neutral ones did no
+ # work worth a check on the pull request.
continue
state, description = JOB_STATUSES[outcome]
context = contexts.setdefault(
@@ -1928,7 +1996,7 @@ class JobStatusReporter:
client: GitHubClient,
*,
repo: str,
- budget: int = MAX_JOB_STATUSES_PER_RUN,
+ budget: int = compute_job_status_budget(None),
now: datetime | None = None,
read_jobs: ReadJobs | None = None,
):
@@ -1968,7 +2036,7 @@ class JobStatusReporter:
The key is recorded in the marker, so a completed run's jobs are read
once. A run in
progress, or one whose statuses did not all fit in the budget, is read
again next time. The
key holds the run attempt, so a re-run in the fork is read again even
when it ends with the
- same conclusion.
+ same conclusion, and the run's passed jobs, which later runs show
without reading the jobs.
"""
if fork.run_id is None or not pr.head_repo:
return None
@@ -1978,8 +2046,14 @@ class JobStatusReporter:
else None
)
recorded = pr.marker.jobs if pr.marker is not None and pr.marker.sha
== pr.head_sha else None
- if key is not None and key == recorded:
- return key
+ # Recorded before the job counts were, a key has none; its jobs are
then read once more.
+ if (
+ key is not None
+ and recorded is not None
+ and recorded.startswith(f"{key}:")
+ and parse_job_counts(recorded) is not None
+ ):
+ return recorded
# Without writes left, the reads are spent for nothing: a backlog of
hundreds of pull requests
# would otherwise cost two reads each on every run.
if self.remaining == 0:
@@ -2004,12 +2078,29 @@ class JobStatusReporter:
)
}
desired = build_job_statuses(jobs)
+ counts = count_jobs(jobs)
+ cancelled_state, cancelled_description =
cancelled_jobs_status(counts.cancelled)
+ desired[CANCELLED_JOBS_CONTEXT] = (cancelled_state, fork.run_url or
"", cancelled_description)
if key is not None:
- # Jobs of an earlier run of this commit that the completed run did
not run, or skipped.
- wanted = {context.casefold() for context in desired}
+ # Jobs of an earlier run of this commit that the completed run did
not run, or skipped. A
+ # cancelled job keeps what an earlier run reported for it.
+ wanted = {context.casefold() for context in desired} | {
+ job_status_context(job["name"]).casefold()
+ for job in jobs
+ if job.get("conclusion") == "cancelled" and job.get("status")
not in RUNNING_JOB_STATUSES
+ }
for folded, (context, (_, target_url, _)) in current.items():
if context.startswith(JOB_STATUS_CONTEXT_PREFIX) and folded
not in wanted:
desired[context] = (STALE_JOB_STATUS[0], target_url,
STALE_JOB_STATUS[1])
+ # Passed jobs are only counted in `Fork CI / Tests (AMD)`: a
full-matrix run has about 130, and
+ # each status is a write. A success is posted only over a failure from
an earlier run of the
+ # commit, which would otherwise keep showing; so is the cancelled
jobs' one.
+ desired = {
+ context: status
+ for context, status in desired.items()
+ if status[0] != "success"
+ or current.get(context.casefold(), ("", ("success", "",
"")))[1][0] != "success"
+ }
changes = sorted(
(
(context, status)
@@ -2025,7 +2116,9 @@ class JobStatusReporter:
self._client.set_status(pr.number, pr.head_sha, state, target_url,
description, context=context)
# Charged per write, so writes made before a later one fails still
count.
self.remaining -= 1
- return key
+ if key is None:
+ return None
+ return f"{key}:{counts.passed}:{counts.cancelled}"
def fetch_workflow_state(client: GitHubClient, fork: str) -> str:
@@ -2721,7 +2814,6 @@ def drop_reported(
attention: list[tuple[int, str]],
notes: list[str],
record: bool = True,
- deferred_runs: int = 0,
rate_limited_runs: int = 0,
job_status_failure_runs: dict[int, int] | None = None,
) -> tuple[list[tuple[int, str]], list[tuple[int, str]], list[str]]:
@@ -2731,7 +2823,6 @@ def drop_reported(
Only the previous run's items are kept, so an item that went away and
comes back is posted
again. A missing or unreadable file reports everything.
- :param deferred_runs: the consecutive runs that deferred job statuses,
recorded for the next run.
:param rate_limited_runs: the consecutive runs GitHub rate-limited,
recorded for the next run.
:param job_status_failure_runs: per pull request, the consecutive runs
whose job statuses failed,
recorded for the next run.
@@ -2740,7 +2831,6 @@ def drop_reported(
"unknown": [list(item) for item in unknown],
"attention": [list(item) for item in attention],
"notes": notes,
- "deferred_runs": deferred_runs,
"rate_limited_runs": rate_limited_runs,
"job_status_failure_runs": {
str(number): runs for number, runs in (job_status_failure_runs or
{}).items()
@@ -2777,6 +2867,55 @@ def read_run_count(reported_file: Path | None, key: str)
-> int | None:
return count if isinstance(count, int) and not isinstance(count, bool)
else 0
+def find_time_since_previous_monitor_run(
+ client: GitHubClient, repo: str, *, run_url: str
+) -> timedelta | None:
+ """
+ The time between the previous monitor run's start and this run's; None
when it cannot be told.
+
+ Both starts are read from GitHub, so they are the same boundary of each
run, setup included, and a
+ failed run or a failed Slack post, which leave the record the monitor
carries between runs
+ unchanged, still counts. A re-run of this run counts from its previous
attempt's start.
+ """
+ match = re.search(r"/actions/runs/(\d+)", run_url)
+ if match is None:
+ return None
+ run_id = int(match[1])
+ try:
+ runs = client.rest_get(
+
f"repos/{repo}/actions/workflows/{MONITOR_WORKFLOW_FILE}/runs?per_page={MONITOR_RUNS_READ}"
+ )["workflow_runs"]
+ except Exception as e:
+ console_print(f"[warning]Could not read the monitor's earlier runs:
{e}[/]")
+ return None
+ current = next((run for run in runs if run["id"] == run_id), None)
+ started = parse_github_datetime(current.get("run_started_at")) if current
else None
+ if current is None or started is None:
+ return None
+ # Runs waiting in the concurrency group have not started, so they posted
nothing.
+ boundaries = [
+ start
+ for run in runs
+ if run["id"] != run_id and run.get("status") not in
NOT_STARTED_RUN_STATUSES
+ for start in [parse_github_datetime(run.get("run_started_at"))]
+ if start is not None and start <= started
+ ]
+ attempt = current.get("run_attempt") or 1
+ if attempt > 1:
+ try:
+ previous_attempt =
client.rest_get(f"repos/{repo}/actions/runs/{run_id}/attempts/{attempt - 1}")
+ except Exception as e:
+ console_print(f"[warning]Could not read the previous attempt of
this run: {e}[/]")
+ return None
+ previous_start =
parse_github_datetime(previous_attempt.get("run_started_at"))
+ if previous_start is None:
+ return None
+ boundaries.append(previous_start)
+ if not boundaries:
+ return None
+ return started - max(boundaries)
+
+
def read_job_status_failure_runs(reported_file: Path | None) -> dict[int, int]:
"""Per pull request, the consecutive runs whose job statuses failed, as
the previous run recorded."""
if reported_file is None:
@@ -2905,10 +3044,13 @@ def run_reconcile(
f"listing the pull requests with `{LABEL_USE_PROJECT_CI}` failed:
{type(e).__name__}"
)
read_jobs: ReadJobs = {}
+ job_status_budget = compute_job_status_budget(
+ find_time_since_previous_monitor_run(client, repo, run_url=run_url)
+ )
job_reporter = JobStatusReporter(
client,
repo=repo,
- budget=max(MAX_JOB_STATUSES_PER_RUN - ci_run_started_writes -
ci_passed_reserved, 0),
+ budget=max(job_status_budget - ci_run_started_writes -
ci_passed_reserved, 0),
now=now,
read_jobs=read_jobs,
)
@@ -2964,6 +3106,7 @@ def run_reconcile(
verify_head_repo_unavailable=functools.partial(
check_head_repo_unavailable, client, repo, pr.number
),
+ job_counts=find_job_counts(fork, read_jobs, jobs_reported),
)
# A revocation to `pending` always goes through: it is idempotent
and must not depend on
# `report_project_ci_passed` succeeding later in this run.
@@ -3011,17 +3154,11 @@ def run_reconcile(
attention.extend(job_reporter.delayed)
notes = [bypass_finder.incomplete] if bypass_finder.incomplete else []
notes.extend(note for note in (ci_run_started_note, *ci_passed_notes) if
note)
- previous_deferred_runs = read_run_count(reported_file, "deferred_runs")
- deferred_runs = (previous_deferred_runs or 0) + 1 if job_reporter.deferred
else 0
if job_reporter.deferred:
console_print(
- f"[info]Job statuses of {len(job_reporter.deferred)} pull requests
left for the next run "
- f"({deferred_runs} runs in a row): {job_reporter.deferred}[/]"
+ f"[info]Job statuses of {len(job_reporter.deferred)} pull requests
left for the next run, "
+ f"after a budget of {job_status_budget} writes:
{job_reporter.deferred}[/]"
)
- # Without a record of earlier runs, deferring is reported at once, as
it cannot be told apart
- # from a budget that does not keep up.
- if previous_deferred_runs is None or deferred_runs >=
DEFERRED_RUNS_BEFORE_ALERT:
- notes.append(JOB_STATUSES_DEFERRED_NOTE)
# The full list for the run log; Slack may show only part of a very long
one.
for number, reason in [*unknown, *attention]:
console_print(f"[warning]#{number}: {reason}[/]")
@@ -3042,7 +3179,6 @@ def run_reconcile(
attention=attention,
notes=notes,
record=not client.dry_run,
- deferred_runs=deferred_runs,
rate_limited_runs=rate_limited_runs,
job_status_failure_runs=job_status_failure_runs,
)
diff --git a/dev/breeze/tests/test_fork_ci_messages.py
b/dev/breeze/tests/test_fork_ci_messages.py
index d945dfe3576..563af5b6723 100644
--- a/dev/breeze/tests/test_fork_ci_messages.py
+++ b/dev/breeze/tests/test_fork_ci_messages.py
@@ -38,9 +38,12 @@ from airflow_breeze.utils.fork_ci_messages import (
MAX_LISTED_FAILED_JOBS,
OPEN_PR_LIMIT_URL,
OVERLAP_LISTED,
+ STATES_WITH_JOB_COUNTS,
BaseCheck,
FailedJob,
+ JobCounts,
Marker,
+ cancelled_jobs_status,
ci_change_approval_comment,
ci_modification_note_comment,
ci_passed_description,
@@ -53,7 +56,9 @@ from airflow_breeze.utils.fork_ci_messages import (
rebase_needed_description,
released_comment,
render_marker,
+ split_job_counts,
state_comment,
+ status_description,
strip_failed_job_list,
tamper_close_comment,
ungated_comment,
@@ -298,10 +303,71 @@ def test_setup_texts_give_the_exact_command(text):
assert "gh workflow enable ci-amd.yml -R u/fork" in text
[email protected](
+ ("state", "job_counts", "expected"),
+ [
+ pytest.param("green", JobCounts(130), "Fork CI passed - see Details
(130 jobs passed)", id="counted"),
+ pytest.param("running", JobCounts(1), "Fork CI running - see Details
(1 job passed)", id="one-job"),
+ pytest.param(
+ "failed",
+ JobCounts(97, 31),
+ "Fork CI failed - see Details (97 jobs passed, 31 cancelled)",
+ id="with-cancelled",
+ ),
+ pytest.param("green", None, "Fork CI passed - see Details",
id="not-counted"),
+ pytest.param(
+ "setup_required",
+ JobCounts(3, 1),
+ "Fork CI not set up - see the pull request comment",
+ id="state-without-a-run",
+ ),
+ ],
+)
+def test_status_description_counts_jobs(state, job_counts, expected):
+ assert status_description(state, job_counts) == expected
+
+
[email protected](
+ ("description", "expected"),
+ [
+ pytest.param(
+ "Fork CI passed - see Details (130 jobs passed)", ("Fork CI passed
- see Details", JobCounts(130))
+ ),
+ pytest.param(
+ "Fork CI running - see Details (1 job passed)", ("Fork CI running
- see Details", JobCounts(1))
+ ),
+ pytest.param(
+ "Fork CI failed - see Details (97 jobs passed, 31 cancelled)",
+ ("Fork CI failed - see Details", JobCounts(97, 31)),
+ ),
+ pytest.param("Fork CI passed - see Details", ("Fork CI passed - see
Details", None)),
+ pytest.param(
+ "Fork CI passed (2 jobs passed) - see Details",
+ ("Fork CI passed (2 jobs passed) - see Details", None),
+ ),
+ ],
+)
+def test_split_job_counts(description, expected):
+ assert split_job_counts(description) == expected
+
+
[email protected](
+ ("cancelled", "expected"),
+ [
+ pytest.param(31, ("failure", "31 jobs cancelled in the fork - see
Details"), id="cancelled"),
+ pytest.param(1, ("failure", "1 job cancelled in the fork - see
Details"), id="one"),
+ pytest.param(0, ("success", "No jobs cancelled in the latest fork
run"), id="none"),
+ ],
+)
+def test_cancelled_jobs_status(cancelled, expected):
+ assert cancelled_jobs_status(cancelled) == expected
+
+
def test_status_descriptions_fit_github_limit():
base_ref = "providers-cncf-kubernetes/v10-22"
descriptions = [
*_STATUS_DESCRIPTIONS.values(),
+ *(status_description(state, JobCounts(99999, 99999)) for state in
STATES_WITH_JOB_COUNTS),
CI_PASSED_REVOKED_DESCRIPTION,
ci_passed_description(BaseCheck("d" * 40, 199), base_ref=base_ref),
*(
diff --git a/dev/breeze/tests/test_fork_ci_reconcile.py
b/dev/breeze/tests/test_fork_ci_reconcile.py
index 4e2ddd2bf90..55246b5c0a3 100644
--- a/dev/breeze/tests/test_fork_ci_reconcile.py
+++ b/dev/breeze/tests/test_fork_ci_reconcile.py
@@ -31,6 +31,7 @@ from airflow_breeze.commands.ci_commands import
fork_ci_reconcile
from airflow_breeze.utils.fork_ci_gate import GateConfig
from airflow_breeze.utils.fork_ci_github import GitHubClient, GitHubError
from airflow_breeze.utils.fork_ci_messages import (
+ CANCELLED_JOBS_CONTEXT,
CI_PASSED_CONTEXT,
CI_PASSED_PROJECT_DESCRIPTION,
CI_PASSED_WAITING_PROJECT_CI_DESCRIPTION,
@@ -42,18 +43,17 @@ from airflow_breeze.utils.fork_ci_messages import (
STATUS_CONTEXT,
BaseCheck,
FailedJob,
+ JobCounts,
Marker,
parse_marker,
render_marker,
)
from airflow_breeze.utils.fork_ci_reconcile import (
- DEFERRED_RUNS_BEFORE_ALERT,
GATE_PROTECTED_PATHS,
JOB_STATUS_FAILURE_RUNS_BEFORE_ALERT,
- JOB_STATUSES_DEFERRED_NOTE,
JOB_STATUSES_FAILED_REASON,
MAX_CI_RUN_STARTED_PER_RUN,
- MAX_JOB_STATUSES_PER_RUN,
+ MAX_JOB_STATUS_BUDGET_INTERVAL,
MAX_PROJECT_CI_PASSED_PER_RUN,
MONITOR_INTERVAL,
RATE_LIMITED_NOTE,
@@ -77,7 +77,9 @@ from airflow_breeze.utils.fork_ci_reconcile import (
check_head_repo_unavailable,
classify,
compare_with_base,
+ compute_job_status_budget,
compute_projection,
+ count_jobs,
discover,
drop_reported,
fetch_fork_observations,
@@ -86,10 +88,12 @@ from airflow_breeze.utils.fork_ci_reconcile import (
find_ci_run_since,
find_failed_jobs,
find_heads_shared_across_bases,
+ find_job_counts,
find_project_ci_label_heads,
find_project_ci_passed_url,
find_project_ci_run_url,
find_state_comment,
+ find_time_since_previous_monitor_run,
is_cancelled_gate_run,
is_gated_skip_run,
is_own_pilot_change,
@@ -108,6 +112,7 @@ from airflow_breeze.utils.fork_ci_reconcile import (
report_project_ci_passed,
report_project_ci_runs_started,
run_reconcile,
+ status_state,
)
from airflow_breeze.utils.path_utils import AIRFLOW_ROOT_PATH
@@ -655,6 +660,20 @@ def test_green_records_the_base_it_tested(pr_overrides,
compared, expected_base)
),
pytest.param(None, _marker("green"), ForkState.GREEN, True,
id="marker-is-not-trusted"),
pytest.param(("success", RUN_URL, "forged"), None, ForkState.GREEN,
True, id="unknown-status"),
+ pytest.param(
+ ("success", RUN_URL, "Fork CI passed - see Details (130 jobs
passed)"),
+ None,
+ ForkState.GREEN,
+ False,
+ id="status-green-with-passed-jobs",
+ ),
+ pytest.param(
+ (UNVERIFIED_STATUS[0], RUN_URL, f"{UNVERIFIED_STATUS[2]} (1 job
passed)"),
+ None,
+ ForkState.GREEN_UNVERIFIED,
+ False,
+ id="status-unverified-with-passed-jobs",
+ ),
],
)
def test_green_trust_is_decided_once_per_commit(current_status, marker,
expected_state, lists_files):
@@ -699,6 +718,96 @@ def
test_green_is_not_trusted_for_a_head_shared_across_bases():
changes.assert_not_called()
[email protected](
+ ("status", "expected"),
+ [
+ pytest.param(
+ ("success", RUN_URL, "Fork CI passed - see Details (12 jobs
passed)"), "green", id="green"
+ ),
+ pytest.param(
+ ("pending", RUN_URL, "Fork CI running - see Details (1 job
passed)"), "running", id="running"
+ ),
+ pytest.param(
+ ("error", RUN_URL, "Project CI ran for a gated pull request - see
Details (3 jobs passed)"),
+ None,
+ id="count-on-a-state-without-one",
+ ),
+ pytest.param(
+ ("success", RUN_URL, "Fork CI passed - see Details (many jobs
passed)"), None, id="no-number"
+ ),
+ pytest.param(
+ ("failure", RUN_URL, "Fork CI failed - see Details (97 jobs
passed, 31 cancelled)"),
+ "failed",
+ id="with-cancelled",
+ ),
+ ],
+)
+def test_status_state_reads_job_counts(status, expected):
+ assert status_state(status) == expected
+
+
[email protected](
+ ("current_status", "job_counts", "expected_status"),
+ [
+ pytest.param(
+ None,
+ JobCounts(130),
+ ("success", RUN_URL, "Fork CI passed - see Details (130 jobs
passed)"),
+ id="counted",
+ ),
+ pytest.param(
+ ("success", RUN_URL, "Fork CI passed - see Details (130 jobs
passed)"),
+ None,
+ None,
+ id="count-kept-when-jobs-not-read",
+ ),
+ pytest.param(
+ ("success", RUN_URL, "Fork CI passed - see Details (129 jobs
passed)"),
+ JobCounts(130),
+ ("success", RUN_URL, "Fork CI passed - see Details (130 jobs
passed)"),
+ id="count-updated",
+ ),
+ pytest.param(
+ ("success", RUN_URL, "Fork CI passed - see Details (130 jobs
passed, 2 cancelled)"),
+ None,
+ None,
+ id="cancelled-count-kept",
+ ),
+ ],
+)
+def test_fork_status_shows_the_job_counts(current_status, job_counts,
expected_status):
+ plan = classify(
+ _pr(current_status=current_status),
+ GREEN,
+ config=CONFIG,
+ now=NOW,
+ repo=REPO,
+ find_bypass_run=lambda: None,
+ changes_ci_files=lambda: False,
+ compare_base=lambda: CURRENT_BASE,
+ job_counts=job_counts,
+ )
+
+ assert plan.state == ForkState.GREEN
+ assert plan.status == expected_status
+
+
+def test_fork_status_does_not_carry_a_count_into_another_state():
+ running = ("pending", RUN_URL, "Fork CI running - see Details (40 jobs
passed)")
+ plan = classify(
+ _pr(current_status=running),
+ GREEN,
+ config=CONFIG,
+ now=NOW,
+ repo=REPO,
+ find_bypass_run=lambda: None,
+ changes_ci_files=lambda: False,
+ compare_base=lambda: CURRENT_BASE,
+ )
+
+ assert plan.status == GREEN_STATUS
+
+
def test_find_heads_shared_across_bases():
prs = [
_pr(number=1, head_sha="a", base_ref="main"),
@@ -2790,7 +2899,6 @@ def
test_drop_reported_reports_everything_without_a_readable_previous_run(tmp_pa
"unknown": [[1, "a"]],
"attention": [],
"notes": [],
- "deferred_runs": 0,
"rate_limited_runs": 0,
"job_status_failure_runs": {},
}
@@ -2986,7 +3094,6 @@ def test_build_job_statuses_reports_each_finished_job():
} == {
"Static checks": "success",
"Postgres tests: core / A": "failure",
- "Special tests": "failure",
}
assert statuses[f"{JOB_STATUS_CONTEXT_PREFIX}Static checks"][1] ==
"https://job/Static checks"
@@ -3001,12 +3108,12 @@ def
test_build_job_statuses_reports_the_worst_of_jobs_sharing_a_name():
assert build_job_statuses(jobs)[f"{JOB_STATUS_CONTEXT_PREFIX}Tests"][:2]
== ("failure", "bad")
[email protected]("name", ["Tests (AMD)", "CI passed (fork)", "x\nCI
passed"])
[email protected]("name", ["Tests (AMD)", "CI passed (fork)", "x\nCI
passed", "Cancelled jobs"])
def test_build_job_statuses_cannot_post_the_reconciler_contexts(name):
(context,) = build_job_statuses([_fork_job(name)])
assert context.startswith(JOB_STATUS_CONTEXT_PREFIX)
- assert context not in (STATUS_CONTEXT, CI_PASSED_CONTEXT)
+ assert context not in (STATUS_CONTEXT, CANCELLED_JOBS_CONTEXT,
CI_PASSED_CONTEXT)
assert "\n" not in context
@@ -3030,21 +3137,21 @@ def _reporter_client(client, jobs, current=()):
def test_job_status_reporter_posts_changed_statuses_failures_first(client):
- already = {
- "context": f"{JOB_STATUS_CONTEXT_PREFIX}Static checks",
- "state": "success",
- "target_url": "https://job/Static checks",
- "description": "Passed in the fork - see Details",
+ failed_before = {
+ "context": f"{JOB_STATUS_CONTEXT_PREFIX}Helm tests",
+ "state": "failure",
+ "target_url": "https://job/Helm tests",
+ "description": "Failed in the fork - see Details",
}
_reporter_client(
client,
[_fork_job("Static checks"), _fork_job("Helm tests"),
_fork_job("Unit", conclusion="failure")],
- [already],
+ [failed_before],
)
key = JobStatusReporter(client, repo=REPO).report(_pr(), COMPLETED_RUN)
- assert key == "1:1:FAILURE"
+ assert key == "1:1:FAILURE:2:0"
assert [(c.args[2], c.kwargs["context"]) for c in
client.set_status.call_args_list] == [
("failure", f"{JOB_STATUS_CONTEXT_PREFIX}Unit"),
("success", f"{JOB_STATUS_CONTEXT_PREFIX}Helm tests"),
@@ -3052,7 +3159,7 @@ def
test_job_status_reporter_posts_changed_statuses_failures_first(client):
def
test_job_status_reporter_leaves_what_does_not_fit_in_the_budget_for_the_next_run(client):
- _reporter_client(client, [_fork_job("A", conclusion="failure"),
_fork_job("B"), _fork_job("C")])
+ _reporter_client(client, [_fork_job(name, conclusion="failure") for name
in "ABC"])
reporter = JobStatusReporter(client, repo=REPO, budget=2)
assert reporter.report(_pr(), COMPLETED_RUN) is None
@@ -3065,7 +3172,9 @@ def
test_job_status_reporter_leaves_what_does_not_fit_in_the_budget_for_the_next
def test_job_status_reporter_does_not_record_a_run_in_progress(client):
- _reporter_client(client, [_fork_job("A"), _fork_job("B",
status="in_progress", conclusion=None)])
+ _reporter_client(
+ client, [_fork_job("A", conclusion="failure"), _fork_job("B",
status="in_progress", conclusion=None)]
+ )
assert JobStatusReporter(client, repo=REPO).report(_pr(), RUNNING_RUN) is
None
client.set_status.assert_called_once()
@@ -3073,19 +3182,31 @@ def
test_job_status_reporter_does_not_record_a_run_in_progress(client):
@pytest.mark.parametrize(
- ("marker_jobs", "marker_sha", "reads"),
+ ("marker_jobs", "marker_sha", "reads", "expected"),
[
- pytest.param("1:1:FAILURE", "abc123", False, id="already-reported"),
- pytest.param("1:1:SUCCESS", "abc123", True,
id="re-run-with-another-conclusion"),
- pytest.param("1:2:FAILURE", "abc123", True,
id="another-attempt-with-the-same-conclusion"),
- pytest.param("1:1:FAILURE", "other", True,
id="marker-of-another-commit"),
+ pytest.param("1:1:FAILURE:7:1", "abc123", False, "1:1:FAILURE:7:1",
id="already-reported"),
+ pytest.param("1:1:FAILURE", "abc123", True, "1:1:FAILURE:0:0",
id="recorded-before-the-counts"),
+ pytest.param(
+ "1:1:FAILURE:7", "abc123", True, "1:1:FAILURE:0:0",
id="recorded-with-the-passed-count-only"
+ ),
+ pytest.param(
+ "1:1:SUCCESS:7:1", "abc123", True, "1:1:FAILURE:0:0",
id="re-run-with-another-conclusion"
+ ),
+ pytest.param(
+ "1:2:FAILURE:7:1",
+ "abc123",
+ True,
+ "1:1:FAILURE:0:0",
+ id="another-attempt-with-the-same-conclusion",
+ ),
+ pytest.param("1:1:FAILURE:7:1", "other", True, "1:1:FAILURE:0:0",
id="marker-of-another-commit"),
],
)
-def test_job_status_reporter_reads_a_completed_run_once(client, marker_jobs,
marker_sha, reads):
+def test_job_status_reporter_reads_a_completed_run_once(client, marker_jobs,
marker_sha, reads, expected):
_reporter_client(client, [])
pr = _pr(marker=dataclasses.replace(_marker("failed", sha=marker_sha),
jobs=marker_jobs))
- assert JobStatusReporter(client, repo=REPO).report(pr, COMPLETED_RUN) ==
"1:1:FAILURE"
+ assert JobStatusReporter(client, repo=REPO).report(pr, COMPLETED_RUN) ==
expected
assert client.rest_get.called is reads
@@ -3195,7 +3316,7 @@ def
test_job_status_reporter_leaves_other_contexts_alone(client):
def
test_job_status_reporter_charges_each_write_even_when_a_later_one_fails(client):
- _reporter_client(client, [_fork_job("A", conclusion="failure"),
_fork_job("B"), _fork_job("C")])
+ _reporter_client(client, [_fork_job(name, conclusion="failure") for name
in "ABC"])
client.set_status.side_effect = [None, GitHubError("HTTP 500")]
reporter = JobStatusReporter(client, repo=REPO, budget=10)
@@ -3226,6 +3347,21 @@ def _reconcile_with_reporter(client, tmp_path,
config=CONFIG, reported_file=None
)
[email protected]("airflow_breeze.utils.fork_ci_reconcile.find_job_counts",
autospec=True)
[email protected]("airflow_breeze.utils.fork_ci_reconcile.JobStatusReporter",
autospec=True)
[email protected]("airflow_breeze.utils.fork_ci_reconcile.classify", autospec=True)
+def test_run_reconcile_shows_the_job_counts(mock_classify, mock_reporter,
mock_counts, client, tmp_path):
+ mock_classify.return_value = Plan(state=ForkState.RUNNING)
+ mock_reporter.return_value.report.return_value = None
+ mock_reporter.return_value.deferred = []
+ mock_reporter.return_value.delayed = []
+ mock_counts.return_value = JobCounts(17, 2)
+
+ _reconcile_with_reporter(client, tmp_path)
+
+ assert mock_classify.call_args.kwargs["job_counts"] == JobCounts(17, 2)
+
+
@mock.patch("airflow_breeze.utils.fork_ci_reconcile.JobStatusReporter",
autospec=True)
@mock.patch("airflow_breeze.utils.fork_ci_reconcile.classify", autospec=True)
def test_run_reconcile_classifies_when_reporting_job_statuses_fails(
@@ -3316,42 +3452,239 @@ def
test_job_status_reporter_does_not_read_without_writes_left(client):
assert reporter.deferred == [42]
[email protected](("deferred", "has_alerts"), [([1], True), ([],
False)])
@mock.patch("airflow_breeze.utils.fork_ci_reconcile.JobStatusReporter",
autospec=True)
@mock.patch("airflow_breeze.utils.fork_ci_reconcile.classify", autospec=True)
-def test_run_reconcile_reports_deferred_job_statuses_to_slack(
- mock_classify, mock_reporter, client, tmp_path, deferred, has_alerts
+def test_run_reconcile_does_not_alert_on_deferred_job_statuses(
+ mock_classify, mock_reporter, client, tmp_path
):
mock_classify.return_value = Plan(state=ForkState.RUNNING)
mock_reporter.return_value.report.return_value = None
- mock_reporter.return_value.deferred = deferred
+ mock_reporter.return_value.deferred = [1]
mock_reporter.return_value.delayed = []
- assert _reconcile_with_reporter(client, tmp_path) is has_alerts
+ assert _reconcile_with_reporter(client, tmp_path) is False
- if has_alerts:
- assert JOB_STATUSES_DEFERRED_NOTE in (tmp_path / "s.json").read_text()
+
[email protected](
+ ("since_previous_run", "budget"),
+ [
+ pytest.param(None, 20, id="no-previous-run"),
+ pytest.param(timedelta(minutes=5), 20, id="on-the-cron"),
+ pytest.param(timedelta(minutes=9, seconds=30), 38, id="started-late"),
+ pytest.param(timedelta(minutes=1), 4,
id="queued-behind-the-previous-run"),
+ pytest.param(timedelta(hours=3), 40, id="after-a-gap"),
+ pytest.param(timedelta(minutes=-1), 0, id="clock-skew"),
+ ],
+)
+def test_compute_job_status_budget(since_previous_run, budget):
+ assert compute_job_status_budget(since_previous_run) == budget
+
+
+def test_job_status_budget_after_a_gap_is_capped():
+ assert compute_job_status_budget(MAX_JOB_STATUS_BUDGET_INTERVAL) ==
compute_job_status_budget(
+ MAX_JOB_STATUS_BUDGET_INTERVAL * 2
+ )
+
+
+MONITOR_RUN_URL = "https://github.com/apache/airflow/actions/runs/100"
+def _monitor_run(run_id, started, status="completed", attempt=1):
+ return {"id": run_id, "status": status, "run_started_at": started,
"run_attempt": attempt}
+
+
[email protected](
+ ("runs", "expected"),
+ [
+ pytest.param(
+ [
+ _monitor_run(102, "2026-11-10T11:59:00Z", status="pending"),
+ _monitor_run(100, "2026-11-10T11:58:00Z",
status="in_progress"),
+ _monitor_run(99, "2026-11-10T11:51:00Z"),
+ _monitor_run(98, "2026-11-10T11:42:00Z"),
+ ],
+ timedelta(minutes=7),
+ id="previous-started-run",
+ ),
+ pytest.param([_monitor_run(100, "2026-11-10T11:58:00Z")], None,
id="first-run"),
+ pytest.param([_monitor_run(99, "2026-11-10T11:51:00Z")], None,
id="this-run-not-listed"),
+ ],
+)
+def test_find_time_since_previous_monitor_run(client, runs, expected):
+ client.rest_get.return_value = {"workflow_runs": runs}
+
+ assert find_time_since_previous_monitor_run(client, REPO,
run_url=MONITOR_RUN_URL) == expected
+ client.rest_get.assert_called_once_with(
+ f"repos/{REPO}/actions/workflows/fork-ci-monitor.yml/runs?per_page=10"
+ )
+
+
[email protected](
+ ("previous_attempt_start", "expected"),
+ [
+ pytest.param("2026-11-10T12:03:00Z", timedelta(minutes=3),
id="counts-from-the-previous-attempt"),
+ pytest.param("2026-11-10T11:50:00Z", timedelta(minutes=6),
id="later-run-in-between"),
+ ],
+)
+def test_find_time_since_previous_monitor_run_for_a_re_run(client,
previous_attempt_start, expected):
+ def rest_get(path):
+ if path.endswith("/actions/runs/100/attempts/2"):
+ return {"id": 100, "run_attempt": 2, "run_started_at":
previous_attempt_start}
+ return {
+ "workflow_runs": [
+ _monitor_run(100, "2026-11-10T12:06:00Z",
status="in_progress", attempt=3),
+ _monitor_run(99, "2026-11-10T12:00:00Z"),
+ ]
+ }
+
+ client.rest_get.side_effect = rest_get
+
+ assert find_time_since_previous_monitor_run(client, REPO,
run_url=MONITOR_RUN_URL) == expected
+
+
+def
test_find_time_since_previous_monitor_run_when_the_previous_attempt_cannot_be_read(client):
+ client.rest_get.side_effect = [
+ {"workflow_runs": [_monitor_run(100, "2026-11-10T12:06:00Z",
status="in_progress", attempt=2)]},
+ GitHubError("gh api failed: HTTP 502"),
+ ]
+
+ assert find_time_since_previous_monitor_run(client, REPO,
run_url=MONITOR_RUN_URL) is None
+
+
+def test_find_time_since_previous_monitor_run_without_a_run_id(client):
+ assert find_time_since_previous_monitor_run(client, REPO,
run_url="https://run") is None
+ client.rest_get.assert_not_called()
+
+
+def
test_find_time_since_previous_monitor_run_when_the_runs_cannot_be_read(client):
+ client.rest_get.side_effect = GitHubError("gh api failed: HTTP 502")
+
+ assert find_time_since_previous_monitor_run(client, REPO,
run_url=MONITOR_RUN_URL) is None
+
+
[email protected]("airflow_breeze.utils.fork_ci_reconcile.find_time_since_previous_monitor_run",
autospec=True)
@mock.patch("airflow_breeze.utils.fork_ci_reconcile.JobStatusReporter",
autospec=True)
[email protected]("airflow_breeze.utils.fork_ci_reconcile.report_project_ci_runs_started",
autospec=True)
@mock.patch("airflow_breeze.utils.fork_ci_reconcile.classify", autospec=True)
-def
test_run_reconcile_reports_deferred_job_statuses_only_when_deferring_persists(
- mock_classify, mock_reporter, client, tmp_path
+def
test_run_reconcile_budgets_job_statuses_for_the_time_since_the_previous_run(
+ mock_classify, mock_report, mock_reporter, mock_previous, client, tmp_path
):
mock_classify.return_value = Plan(state=ForkState.RUNNING)
+ mock_report.return_value = []
mock_reporter.return_value.report.return_value = None
- client.dry_run = False
- reported_file = tmp_path / "reported.json"
- reported_file.write_text("{}")
- alerts = []
- for deferred in ([1], [1], [], *[[1]] * DEFERRED_RUNS_BEFORE_ALERT):
- mock_reporter.return_value.deferred = deferred
- mock_reporter.return_value.delayed = []
- alerts.append(_reconcile_with_reporter(client, tmp_path,
reported_file=reported_file))
+ mock_reporter.return_value.deferred = []
+ mock_reporter.return_value.delayed = []
+ mock_previous.return_value = timedelta(minutes=9)
+
+ _reconcile_with_reporter(client, tmp_path)
+
+ assert mock_reporter.call_args.kwargs["budget"] ==
compute_job_status_budget(timedelta(minutes=9))
+
+
+def test_count_jobs():
+ jobs = [
+ _fork_job("A"),
+ _fork_job("B"),
+ _fork_job("C", conclusion="failure"),
+ _fork_job("D", conclusion="skipped"),
+ _fork_job("E", status="in_progress", conclusion=None),
+ _fork_job("F", conclusion="cancelled"),
+ _fork_job("G", status="in_progress", conclusion="cancelled"),
+ _fork_job("Platform: AMD"),
+ _fork_job("CI passed", conclusion="cancelled"),
+ ]
+
+ assert count_jobs(jobs) == JobCounts(passed=2, cancelled=1)
+
+
[email protected](
+ ("fork", "read_jobs", "jobs_reported", "expected"),
+ [
+ pytest.param(COMPLETED_RUN, {1: [_fork_job("A")]}, None, JobCounts(1),
id="read"),
+ pytest.param(
+ COMPLETED_RUN, {1: [_fork_job("A")]}, "1:1:FAILURE:9:3",
JobCounts(1), id="read-wins-over-the-key"
+ ),
+ pytest.param(COMPLETED_RUN, {}, "1:1:FAILURE:9:3", JobCounts(9, 3),
id="from-the-recorded-key"),
+ pytest.param(COMPLETED_RUN, {}, "1:1:FAILURE", None,
id="key-recorded-before-the-counts"),
+ pytest.param(COMPLETED_RUN, {}, "1:1:FAILURE:9:x", None,
id="malformed-key"),
+ pytest.param(COMPLETED_RUN, {}, None, None, id="not-read"),
+ pytest.param(COMPLETED_RUN, {1: None}, None, None, id="read-failed"),
+ pytest.param(ForkObservation(), {1: [_fork_job("A")]},
"1:1:FAILURE:9:3", None, id="no-run"),
+ ],
+)
+def test_find_job_counts(fork, read_jobs, jobs_reported, expected):
+ assert find_job_counts(fork, read_jobs, jobs_reported) == expected
+
+
+def
test_job_status_reporter_counts_cancelled_jobs_instead_of_posting_them(client):
+ failed_before = {
+ "context": f"{JOB_STATUS_CONTEXT_PREFIX}Helm tests",
+ "state": "failure",
+ "target_url": "https://job/Helm tests",
+ "description": "Failed in the fork - see Details",
+ }
+ _reporter_client(
+ client,
+ [
+ _fork_job("Unit", conclusion="failure"),
+ _fork_job("Helm tests", conclusion="cancelled"),
+ _fork_job("MySQL tests", conclusion="cancelled"),
+ ],
+ [failed_before],
+ )
+
+ key = JobStatusReporter(client, repo=REPO).report(_pr(), COMPLETED_RUN)
+
+ assert key == "1:1:FAILURE:0:2"
+ # One status for the cancelled jobs; the earlier failure of a job the run
cancelled is kept rather
+ # than marked as not in the run.
+ assert [c.args[2:] + (c.kwargs["context"],) for c in
client.set_status.call_args_list] == [
+ ("failure", RUN_URL, "2 jobs cancelled in the fork - see Details",
CANCELLED_JOBS_CONTEXT),
+ (
+ "failure",
+ "https://job/Unit",
+ "Failed in the fork - see Details",
+ f"{JOB_STATUS_CONTEXT_PREFIX}Unit",
+ ),
+ ]
+
+
[email protected](
+ ("current", "posted"),
+ [
+ pytest.param((), [], id="nothing-cancelled-before"),
+ pytest.param(
+ [
+ {
+ "context": CANCELLED_JOBS_CONTEXT,
+ "state": "failure",
+ "target_url":
"https://github.com/someone/airflow/actions/runs/0",
+ "description": "3 jobs cancelled in the fork - see
Details",
+ }
+ ],
+ [("success", RUN_URL, "No jobs cancelled in the latest fork run")],
+ id="cleared-after-a-re-run",
+ ),
+ pytest.param(
+ [
+ {
+ "context": CANCELLED_JOBS_CONTEXT,
+ "state": "success",
+ "target_url":
"https://github.com/someone/airflow/actions/runs/0",
+ "description": "No jobs cancelled in the latest fork run",
+ }
+ ],
+ [],
+ id="already-cleared",
+ ),
+ ],
+)
+def
test_job_status_reporter_posts_the_cancelled_jobs_status_only_when_needed(client,
current, posted):
+ _reporter_client(client, [_fork_job("A")], current)
+
+ JobStatusReporter(client, repo=REPO).report(_pr(), COMPLETED_RUN)
- assert alerts == [*[False] * (DEFERRED_RUNS_BEFORE_ALERT + 2), True]
- assert JOB_STATUSES_DEFERRED_NOTE in (tmp_path / "s.json").read_text()
- assert json.loads(reported_file.read_text())["deferred_runs"] ==
DEFERRED_RUNS_BEFORE_ALERT
+ assert [c.args[2:] for c in client.set_status.call_args_list] == posted
def test_build_job_statuses_treats_contexts_case_insensitively():
@@ -3373,7 +3706,7 @@ def
test_job_status_reporter_compares_contexts_case_insensitively(client):
}
_reporter_client(client, [_fork_job("Tests", conclusion="failure")],
[posted])
- assert JobStatusReporter(client, repo=REPO).report(_pr(), COMPLETED_RUN)
== "1:1:FAILURE"
+ assert JobStatusReporter(client, repo=REPO).report(_pr(), COMPLETED_RUN)
== "1:1:FAILURE:0:0"
client.set_status.assert_not_called()
@@ -3998,8 +4331,8 @@ def
test_run_reconcile_retries_a_pull_request_whose_head_moved(client, tmp_path)
@pytest.mark.parametrize(
("fails", "job_status_budget"),
[
- pytest.param(False, MAX_JOB_STATUSES_PER_RUN - 2, id="reported"),
- pytest.param(True, MAX_JOB_STATUSES_PER_RUN -
MAX_CI_RUN_STARTED_PER_RUN, id="fails"),
+ pytest.param(False, compute_job_status_budget(None) - 2,
id="reported"),
+ pytest.param(True, compute_job_status_budget(None) -
MAX_CI_RUN_STARTED_PER_RUN, id="fails"),
],
)
@mock.patch("airflow_breeze.utils.fork_ci_reconcile.JobStatusReporter",
autospec=True)