This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v3-4-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/v3-4-test by this push:
new bd81ce18777 [v3-4-test] Harden fork CI reconciler state transitions
and closure rules (#76956) (#77124)
bd81ce18777 is described below
commit bd81ce187772921721048e25bebd5750f36bc528
Author: github-actions[bot]
<41898282+github-actions[bot]@users.noreply.github.com>
AuthorDate: Sun Nov 15 16:10:20 2026 +0100
[v3-4-test] Harden fork CI reconciler state transitions and closure rules
(#76956) (#77124)
Include PullRequestReview and discussion comments in activity tracking so
diff comments and reviews count as human activity, reset the inactivity clock
when a new commit SHA is pushed, and verify head repository availability via
the REST API before reporting deleted forks as unknown. Case-fold bot logins so
triage bots do not prevent inactive pull requests from closing, sort
workflow_dispatch check suites by run attempt, separate timeline control events
from comments/reviews in the di [...]
(cherry picked from commit febd7b45a0979a11b8a37d761ac8da6c520a09c5)
Co-authored-by: Ayush Meel <[email protected]>
---
.../doc/ci/08_fork_ci_for_external_contributors.md | 33 +--
.../src/airflow_breeze/utils/fork_ci_messages.py | 9 +-
.../src/airflow_breeze/utils/fork_ci_reconcile.py | 136 ++++++++--
dev/breeze/tests/test_fork_ci_reconcile.py | 288 ++++++++++++++++++++-
4 files changed, 423 insertions(+), 43 deletions(-)
diff --git a/dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md
b/dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md
index d74f2e6a751..cfee5eae88a 100644
--- a/dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md
+++ b/dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md
@@ -575,7 +575,7 @@ one of their other open pull requests first.
| released | Usual review with project CI | undraft, except the author's draft
| unchanged | remove |
| ungated | Usual review with project CI | undraft, except the author's draft
| unchanged | remove |
| closed | Author sets up fork CI and reopens | unchanged | remove | remove |
-| gate_modified | Maintainer reopens and adds `use project ci` | unchanged |
remove | remove |
+| gate_modified | Author restores the gate and reopens, or maintainer adds
`use project ci` | unchanged | remove | remove |
| gate_bypassed | Maintainer reopens and adds `use project ci` | unchanged |
remove | remove |
The reconciler never undrafts the author's draft. It does not draft a closed
pull request either,
@@ -589,7 +589,7 @@ since the draft-state job runs after the close and GitHub
refuses to draft a clo
| **failed**: conclusion is failure, cancelled, timed out or startup failure |
`failure` (red), link to the run; for a startup failure the description says
the run could not start | update with the run link and the jobs that did not
pass; for a startup failure the text says no job ran and points to the fork's
Actions settings |
| **green**: conclusion success and the pull request does not change
CI-defining files | `success` (green), link to the run | update: ready for
review, run link; on the author's draft: mark it ready for review when it is |
| **green_unverified**: conclusion success, but the pull request changes
CI-defining files (see "Trusting a green fork run") | `pending` (yellow), link
to the run | update: a maintainer must run project CI; on the author's draft:
mark it ready for review, then a maintainer runs project CI |
-| **closed**: setup_required or awaiting_push for `close_after_days` |
unchanged | close with a new comment; reopening restarts the clock |
+| **closed**: setup_required or awaiting_push for `close_after_days` without
human comments | unchanged | close with a new comment; reopening restarts the
clock |
| **released**: the pull request carries `use project ci` | `success` (green),
link to the pull request's checks | update: project CI is in use; `fork ci
required` is removed |
| **ungated**: the base branch has no gate, for example after retargeting to
an older release branch | `success` (green), link to the pull request's checks
| update: project CI is in use; `fork ci required` is removed |
| **gate_modified**: the pull request changes the fork CI gate (see "Gate
protection") | `error` (red), link to the pull request's changed files | post a
**new** comment explaining why, then close; checked after gate tampering,
before the fork state |
@@ -617,12 +617,16 @@ Rules that sit under the table:
are touched, and adding `use project ci` takes a pull request out of the
state machine on the
next run (see "Override labels"). Removing `fork ci required` by hand is not
enough: it updates
the pull request, so discovery finds it again and labels it back while it is
still gated.
-- **Deleted head repository** is reported as unknown (see below) and otherwise
left alone.
+- **Deleted head repository** is verified via the GitHub REST API to rule out
transient API
+ errors. If confirmed unavailable, it is reported as unknown (see below) and
otherwise left alone.
- **The close clock** starts at the `since` timestamp recorded for the current
state and
- resets on any state change. Closing for inactivity records `closed` in the
marker, so the
- first run after a reopen sees a state change from `closed` and starts a
fresh clock rather
- than closing the pull request again at once. Only a reopen recorded in the
timeline after
- the marker counts: an open pull request with a `closed` marker and no such
reopen is one whose
+ resets on any state change or new commit push. Before closing after
`close_after_days` (in
+ `setup_required` or `awaiting_push`), the reconciler checks for recent human
activity
+ (including issue comments, reviews, inline diff comments, and suggestions);
closure is deferred
+ as long as human discussion has occurred within `close_after_days`. Closing
for inactivity records
+ `closed` in the marker, so the first run after a reopen sees a state change
from `closed` and starts
+ a fresh clock rather than closing the pull request again at once. Only a
reopen recorded in the timeline
+ after the marker counts: an open pull request with a `closed` marker and no
such reopen is one whose
closing comment or close failed, and the next run finishes the closure. The
same holds for a
`gate_bypassed` marker, where only a reopen by someone other than the author
counts, so the
closure is finished even after the offending run has left the tampering
check's window.
@@ -815,10 +819,9 @@ released to project CI.
The state is `gate_modified`: the pull request loses `ready for maintainer
review`
and `needs project ci`, gets an `error` status linking its changed files, and
the closure is
-reported to Slack as an attention item. Gate tampering is checked first and
wins. As for gate
-tampering, only a reopen by someone other than the author counts: an open pull
request whose
-marker says `gate_modified` and that was not reopened since is closed again,
and a maintainer's
-reopen without restoring the gate closes it again on the next run. A
maintainer who wants the
+reported to Slack as an attention item. Gate tampering is checked first and
wins. When closed for `gate_modified`,
+the author can revert or remove the gate change and reopen the pull request;
if reopened with the
+gate change still present, it is closed again on the next run. A maintainer
who wants the
change adds `use project ci`, which releases the pull request before any of
these checks. Each
of these files carries a comment saying so.
@@ -832,9 +835,9 @@ an HTML marker the reconciler parses:
```
followed by the human text for the current state. Editing does not notify,
which is the
-point: state churn is silent. The three moments that need the author's
attention, the nudge
-after enabling workflows, the close after inactivity and the close for gate
tampering, are
-new comments.
+point: state churn is silent. The moments that need the author's attention,
such as the nudge
+after enabling workflows, the close after inactivity and the close for
+gate tampering, are new comments.
All comment texts live in
`dev/breeze/src/airflow_breeze/utils/fork_ci_messages.py` as
templates so wording can be reviewed in one place. Every comment ends with the
AI-attribution
@@ -1213,7 +1216,7 @@ Unknown means any of:
conclusion outside the modelled set.
- A workflow state outside `active` / `disabled_fork` / `disabled_manually` /
`disabled_inactivity` / not found.
- A mutation that returned an error (label, comment, close, status).
-- A head repository that no longer exists.
+- A head repository that no longer exists (after confirming via the REST API
to rule out transient API errors).
- A state comment whose marker cannot be parsed.
Expected states never alert. The message lists the pull request numbers and
the reason for
diff --git a/dev/breeze/src/airflow_breeze/utils/fork_ci_messages.py
b/dev/breeze/src/airflow_breeze/utils/fork_ci_messages.py
index 9d852c7a9fc..7378ad7f368 100644
--- a/dev/breeze/src/airflow_breeze/utils/fork_ci_messages.py
+++ b/dev/breeze/src/airflow_breeze/utils/fork_ci_messages.py
@@ -353,7 +353,7 @@ def is_fork_ci_passed_description(description: str | None)
-> bool:
def state_comment(
marker: Marker,
*,
- fork: str,
+ fork: str | None = None,
run_url: str | None,
base_ref: str = "main",
author_draft: bool = False,
@@ -374,7 +374,7 @@ def state_comment(
parts.append(f"Latest run: {run_url}")
if marker.state == "failed" and failed_jobs:
parts.append(build_failed_job_list(failed_jobs))
- if marker.state == "setup_required":
+ if marker.state == "setup_required" and fork:
parts.append(_setup_instructions(fork))
parts.append(f"How this works: [CI in your fork]({CONTRIBUTING_URL}).")
return "\n\n".join(parts) + FOOTER
@@ -398,8 +398,9 @@ def gate_modified_close_comment(*, change: str | None) ->
str:
"monitor, and a pull request whose CI runs in a fork must not change
it, because the change would "
"decide where its own CI runs. The one exception is adding, changing
or removing your own `[pilot]` "
"entry in `.github/fork-ci-gate.toml`.\n\n"
- "If the change is wanted, a maintainer can reopen the pull request and
add the `use project ci` "
- "label." + FOOTER
+ "If the change was accidental, you can revert or remove the change to
the gate files and reopen "
+ "the pull request. If the change is wanted, a maintainer can reopen
the pull request and add the "
+ "`use project ci` label." + FOOTER
)
diff --git a/dev/breeze/src/airflow_breeze/utils/fork_ci_reconcile.py
b/dev/breeze/src/airflow_breeze/utils/fork_ci_reconcile.py
index ed01ce7ad98..61a30366219 100644
--- a/dev/breeze/src/airflow_breeze/utils/fork_ci_reconcile.py
+++ b/dev/breeze/src/airflow_breeze/utils/fork_ci_reconcile.py
@@ -130,7 +130,19 @@ GATED_RUN_JOB_NAMES =
frozenset[str]().union(*GATED_RUN_JOBS)
CHECK_RUN_NEEDED_JOB_NAME = "Check if the run is needed"
PROJECT_CI_NOT_RUN_JOB_NAME = "CI passed (project CI not run)"
CI_PASSED_CANCELLED_JOB_NAME = "CI passed (run cancelled)"
-BOT_LOGINS = frozenset({"github-actions", "github-actions[bot]"})
+BOT_LOGINS = frozenset(
+ {
+ "github-actions",
+ "github-actions[bot]",
+ "airflow-triage-bot",
+ "codecov",
+ "codecov[bot]",
+ "dependabot",
+ "dependabot[bot]",
+ "asf-ci[bot]",
+ "asfbot",
+ }
+)
DISCOVERY_WINDOW = timedelta(hours=24)
PASS_TWO_PAGE_SIZE = 25
FORK_BATCH_SIZE = 50
@@ -303,6 +315,7 @@ class PullRequestSnapshot:
last_reopened_at: datetime | None = None
last_reopened_by_anyone_at: datetime | None = None
last_base_change_at: datetime | None = None
+ last_comment_at: datetime | None = None
# (changed at, previous base, new base) for each retarget, oldest first.
base_changes: tuple[tuple[datetime, str, str], ...] = ()
current_status: tuple[str, str, str] | None = None
@@ -436,7 +449,10 @@ def _workflow_file(resource_path: str | None) -> str:
def _is_bot(login: str | None) -> bool:
- return login in BOT_LOGINS
+ if not login:
+ return False
+ normalized = login.casefold()
+ return normalized in BOT_LOGINS or normalized.endswith("[bot]")
def observe_fork_state(fork: ForkObservation, marker: Marker | None) ->
tuple[ForkState | None, str | None]:
@@ -649,6 +665,7 @@ def classify(
head_shared_across_bases: bool = False,
find_gate_change: Callable[[], str | None] = lambda: None,
find_failed_jobs: Callable[[], list[FailedJob] | None] = lambda: [],
+ verify_head_repo_unavailable: Callable[[], bool] = lambda: False,
) -> Plan:
"""
Decide the mutations for one gated pull request.
@@ -710,6 +727,9 @@ def classify(
return release(ForkState.UNGATED, functools.partial(ungated_comment,
base_ref=pr.base_ref))
if pr.head_repo is None:
+ if not verify_head_repo_unavailable():
+ plan.unknown.append("head repository is unavailable due to GitHub
API error; retrying")
+ return plan
plan.unknown.append("head repository no longer exists")
return plan
@@ -739,7 +759,10 @@ def classify(
reopened_since_closing = {
ForkState.CLOSED.value: pr.last_reopened_by_anyone_at,
ForkState.GATE_BYPASSED.value: pr.last_reopened_at,
- ForkState.GATE_MODIFIED.value: pr.last_reopened_at,
+ ForkState.GATE_MODIFIED.value: max(
+ (t for t in (pr.last_reopened_by_anyone_at, pr.last_reopened_at)
if t is not None),
+ default=None,
+ ),
}
if marker and marker.state in reopened_since_closing:
reopened = reopened_since_closing[marker.state]
@@ -841,7 +864,7 @@ def classify(
"so `CI passed` waits for the next run"
)
plan.state = state
- since = marker.since if marker and marker.state == state.value else now
+ since = marker.since if marker and marker.state == state.value and
marker.sha == pr.head_sha else now
if marker and marker.state == ForkState.SETUP_REQUIRED.value and state ==
ForkState.AWAITING_PUSH:
plan.new_comments.append(nudge_comment(fork=pr.head_repo))
@@ -856,7 +879,14 @@ def classify(
workflow_disabled_comment(fork=pr.head_repo,
workflow_state=fork.workflow_state)
)
- if state in CLOSABLE_STATES and now - since >=
timedelta(days=config.close_after_days):
+ inactive_for_closing = pr.last_comment_at is None or now -
pr.last_comment_at >= timedelta(
+ days=config.close_after_days
+ )
+ if (
+ state in CLOSABLE_STATES
+ and now - since >= timedelta(days=config.close_after_days)
+ and inactive_for_closing
+ ):
_enter_state(plan, pr, labels, ForkState.CLOSED)
_set_marker_comment(
plan,
@@ -867,7 +897,8 @@ def classify(
run_url=None,
),
)
-
plan.new_comments.append(inactivity_close_comment(days=config.close_after_days,
fork=pr.head_repo))
+ comment = inactivity_close_comment(days=config.close_after_days,
fork=pr.head_repo or "")
+ plan.new_comments.append(comment)
plan.close = True
return plan
@@ -965,6 +996,18 @@ fragment PassTwoFields on PullRequest {
... on BaseRefChangedEvent { createdAt previousRefName
currentRefName }
}
}
+ latestComments: comments(last: 10) {
+ nodes { createdAt author { login } }
+ }
+ latestReviews: reviews(last: 10) {
+ nodes {
+ createdAt
+ author { login }
+ comments(last: 10) {
+ nodes { createdAt author { login } }
+ }
+ }
+ }
commits(last: 1) { nodes { commit {
oid
status {
@@ -975,7 +1018,7 @@ fragment PassTwoFields on PullRequest {
} } }
comments(first: 100) {
pageInfo { hasNextPage endCursor }
- nodes { databaseId body author { login } }
+ nodes { databaseId body createdAt author { login } }
}
}
"""
@@ -1044,15 +1087,17 @@ def parse_pull_request(node: dict[str, Any]) ->
PullRequestSnapshot:
last_reopened = None
last_reopened_by_anyone = None
last_base_change = None
+ last_comment_at = None
base_changes = []
for item in (node.get("timelineItems") or {}).get("nodes") or []:
if item["__typename"] == "ConvertToDraftEvent":
last_draft_actor = _login(item.get("actor"))
elif item["__typename"] == "BaseRefChangedEvent":
last_base_change = parse_github_datetime(item["createdAt"])
- base_changes.append(
- (last_base_change, item.get("previousRefName") or "",
item.get("currentRefName") or "")
- )
+ if last_base_change is not None:
+ base_changes.append(
+ (last_base_change, item.get("previousRefName") or "",
item.get("currentRefName") or "")
+ )
elif item["__typename"] == "ReopenedEvent":
last_reopened_by_anyone = parse_github_datetime(item["createdAt"])
# Only a reopen by someone else counts for the gate tampering
check: an author closing
@@ -1064,6 +1109,23 @@ def parse_pull_request(node: dict[str, Any]) ->
PullRequestSnapshot:
if label == LABEL_USE_PROJECT_CI:
last_unlabel = parse_github_datetime(item["createdAt"])
+ for c in (node.get("latestComments") or {}).get("nodes") or []:
+ if not _is_bot(_login(c.get("author"))) and c.get("createdAt"):
+ t = parse_github_datetime(c["createdAt"])
+ if t is not None:
+ last_comment_at = max(filter(None, (last_comment_at, t)),
default=t)
+
+ for r in (node.get("latestReviews") or {}).get("nodes") or []:
+ if not _is_bot(_login(r.get("author"))) and r.get("createdAt"):
+ t = parse_github_datetime(r["createdAt"])
+ if t is not None:
+ last_comment_at = max(filter(None, (last_comment_at, t)),
default=t)
+ for rc in (r.get("comments") or {}).get("nodes") or []:
+ if not _is_bot(_login(rc.get("author"))) and rc.get("createdAt"):
+ t = parse_github_datetime(rc["createdAt"])
+ if t is not None:
+ last_comment_at = max(filter(None, (last_comment_at, t)),
default=t)
+
current_status = None
status_created_at = None
ci_passed_state = None
@@ -1088,6 +1150,10 @@ def parse_pull_request(node: dict[str, Any]) ->
PullRequestSnapshot:
status_created_at = parse_github_datetime(context.get("createdAt")
or "")
comments = node.get("comments") or {}
+ for c in comments.get("nodes") or []:
+ if not _is_bot(_login(c.get("author"))) and c.get("createdAt"):
+ t = parse_github_datetime(c["createdAt"])
+ last_comment_at = max(filter(None, (last_comment_at, t)),
default=t)
found = find_marker_comment(comments.get("nodes") or [])
page_info = comments.get("pageInfo") or {}
comments_cursor = page_info.get("endCursor") if found is None and
page_info.get("hasNextPage") else None
@@ -1113,6 +1179,7 @@ def parse_pull_request(node: dict[str, Any]) ->
PullRequestSnapshot:
last_reopened_at=last_reopened,
last_reopened_by_anyone_at=last_reopened_by_anyone,
last_base_change_at=last_base_change,
+ last_comment_at=last_comment_at,
base_changes=tuple(base_changes),
current_status=current_status,
status_created_at=status_created_at,
@@ -1754,13 +1821,19 @@ def fetch_fork_observations(
suites = [
suite
for suite in (commit.get("checkSuites") or {}).get("nodes") or
[]
- if (run := suite.get("workflowRun") or {}).get("event") ==
"push"
+ if (run := suite.get("workflowRun") or {}).get("event") in
("push", "workflow_dispatch")
and _workflow_file((run.get("workflow") or
{}).get("resourcePath")) == CI_WORKFLOW_FILE
]
if not suites:
observations[pr.number] = ForkObservation()
continue
- latest = max(suites, key=lambda suite: suite["createdAt"])
+ latest = max(
+ suites,
+ key=lambda suite: (
+ suite["createdAt"],
+ (suite.get("workflowRun") or {}).get("runAttempt") or 1,
+ ),
+ )
observations[pr.number] = ForkObservation(
suite_status=latest["status"],
suite_conclusion=latest["conclusion"],
@@ -1998,7 +2071,7 @@ def load_workflow_triggers(workflows_dir: Path) ->
dict[str, frozenset[str]]:
"""Map each workflow file name of a checkout to the events that trigger
it."""
triggers = {}
for path in sorted(workflows_dir.glob("*.y*ml")):
- workflow = yaml.safe_load(path.read_text()) or {}
+ workflow = yaml.safe_load(path.read_text(encoding="utf-8")) or {}
triggers[path.name] = get_workflow_triggers(workflow if
isinstance(workflow, dict) else {})
return triggers
@@ -2148,12 +2221,18 @@ class BypassFinder:
paginate=True,
)
window_start = self._now - TAMPER_WINDOW
- return [
- pull["number"]
- for pull in pulls
- if pull["number"] != pr.number
- and (pull.get("closed_at") is None or
parse_github_datetime(pull["closed_at"]) >= window_start)
- ]
+ active_pulls = []
+ for pull in pulls:
+ if pull["number"] == pr.number:
+ continue
+ closed_at_str = pull.get("closed_at")
+ if closed_at_str is None:
+ active_pulls.append(pull["number"])
+ else:
+ closed_at = parse_github_datetime(closed_at_str)
+ if closed_at is not None and closed_at >= window_start:
+ active_pulls.append(pull["number"])
+ return active_pulls
def find(self, pr: PullRequestSnapshot) -> str | None:
if not pr.head_repo:
@@ -2404,7 +2483,7 @@ query($owner: String!, $name: String!, $number: Int!,
$cursor: String) {
repository(owner: $owner, name: $name) { pullRequest(number: $number) {
comments(first: 100, after: $cursor) {
pageInfo { hasNextPage endCursor }
- nodes { databaseId body author { login } }
+ nodes { databaseId body createdAt author { login } }
}
} }
}
@@ -2419,6 +2498,10 @@ def find_state_comment(client: GitHubClient, repo: str,
pr: PullRequestSnapshot)
comments = client.graphql(
COMMENTS_QUERY, {"owner": owner, "name": name, "number":
pr.number, "cursor": cursor}
)["repository"]["pullRequest"]["comments"]
+ for c in comments.get("nodes") or []:
+ if not _is_bot(_login(c.get("author"))) and c.get("createdAt"):
+ t = parse_github_datetime(c["createdAt"])
+ pr = replace(pr, last_comment_at=max(filter(None,
(pr.last_comment_at, t)), default=t))
found = find_marker_comment(comments["nodes"])
if found is not None:
marker, comment_id, body, error = found
@@ -2509,6 +2592,16 @@ def compare_with_base(client: GitHubClient, repo: str,
pr: PullRequestSnapshot)
return BaseCheck(merge_base, behind, tuple(sorted(overlap)))
+def check_head_repo_unavailable(client: GitHubClient, repo: str, pr_number:
int) -> bool:
+ try:
+ pr_data = client.rest_get(f"repos/{repo}/pulls/{pr_number}")
+ if not isinstance(pr_data, dict):
+ return False
+ return (pr_data.get("head") or {}).get("repo") is None
+ except Exception:
+ return False
+
+
def find_heads_shared_across_bases(prs: Iterable[PullRequestSnapshot]) ->
set[str]:
"""Head commits of more than one pull request whose bases differ."""
bases_by_head: dict[str, set[str]] = {}
@@ -2868,6 +2961,9 @@ def run_reconcile(
head_shared_across_bases=pr.head_sha in shared_heads,
find_gate_change=functools.partial(gate_change_finder.find,
pr),
find_failed_jobs=functools.partial(read_failed_jobs, client,
pr, fork, read_jobs),
+ verify_head_repo_unavailable=functools.partial(
+ check_head_repo_unavailable, client, repo, pr.number
+ ),
)
# A revocation to `pending` always goes through: it is idempotent
and must not depend on
# `report_project_ci_passed` succeeding later in this run.
diff --git a/dev/breeze/tests/test_fork_ci_reconcile.py
b/dev/breeze/tests/test_fork_ci_reconcile.py
index fa1d432977e..4e2ddd2bf90 100644
--- a/dev/breeze/tests/test_fork_ci_reconcile.py
+++ b/dev/breeze/tests/test_fork_ci_reconcile.py
@@ -48,6 +48,7 @@ from airflow_breeze.utils.fork_ci_messages import (
)
from airflow_breeze.utils.fork_ci_reconcile import (
DEFERRED_RUNS_BEFORE_ALERT,
+ GATE_PROTECTED_PATHS,
JOB_STATUS_FAILURE_RUNS_BEFORE_ALERT,
JOB_STATUSES_DEFERRED_NOTE,
JOB_STATUSES_FAILED_REASON,
@@ -68,10 +69,12 @@ from airflow_breeze.utils.fork_ci_reconcile import (
Projection,
PullRequestChangedError,
PullRequestSnapshot,
+ _is_bot,
apply_plan,
build_job_statuses,
build_slack_payload,
changes_ci_definition,
+ check_head_repo_unavailable,
classify,
compare_with_base,
compute_projection,
@@ -156,6 +159,7 @@ def _classify(
base=CURRENT_BASE,
head_shared_across_bases=False,
gate_change=None,
+ verify_head_repo_unavailable=lambda: False,
):
find_bypass = mock.Mock(return_value=bypass_run)
changes = mock.Mock(return_value=changes_ci_files)
@@ -170,6 +174,7 @@ def _classify(
compare_base=mock.Mock(return_value=base),
head_shared_across_bases=head_shared_across_bases,
find_gate_change=lambda: gate_change,
+ verify_head_repo_unavailable=verify_head_repo_unavailable,
)
return plan, find_bypass, changes
@@ -822,6 +827,31 @@ def test_close_clock(marker, should_close):
assert plan.state == ForkState.SETUP_REQUIRED
[email protected](
+ ("marker", "last_comment_at", "should_close"),
+ [
+ pytest.param(_marker("failed", days_ago=7), None, True,
id="failed-seven-days-no-comments"),
+ pytest.param(
+ _marker("failed", days_ago=7), NOW - timedelta(days=8), True,
id="failed-seven-days-old-comments"
+ ),
+ pytest.param(
+ _marker("failed", days_ago=7),
+ NOW - timedelta(days=2),
+ False,
+ id="failed-seven-days-recent-comments",
+ ),
+ pytest.param(_marker("failed", days_ago=6.9), None, False,
id="failed-under-seven-days"),
+ ],
+)
+def test_failing_pull_request_inactivity_clock(marker, last_comment_at,
should_close):
+ pr = _pr(marker=marker, last_comment_at=last_comment_at)
+ failed_fork = ForkObservation(suite_status="COMPLETED",
suite_conclusion="FAILURE")
+ plan, _, _ = _classify(pr, failed_fork)
+
+ assert plan.close is False
+ assert plan.state == ForkState.FAILED
+
+
@pytest.mark.parametrize(
"reopened_at",
[pytest.param(None, id="never-reopened"), pytest.param(NOW -
timedelta(days=31), id="reopened-before")],
@@ -959,12 +989,46 @@ def test_both_override_labels_are_reported():
assert len(plan.attention) == 1
-def test_deleted_head_repository_is_unknown():
- plan, _, _ = _classify(_pr(head_repo=None), GREEN)
+def test_deleted_head_repository_reported_as_unknown():
+ plan, _, _ = _classify(_pr(head_repo=None), GREEN,
verify_head_repo_unavailable=lambda: True)
+ assert plan.state is None
+ assert plan.close is False
assert plan.unknown == ["head repository no longer exists"]
- assert plan.status is None
- assert plan.draft is None
+ assert plan.new_comments == []
+
+
+def test_deleted_head_repository_default_retries_safely():
+ plan, _, _ = _classify(_pr(head_repo=None), GREEN)
+
+ assert plan.state is None
+ assert plan.close is False
+ assert plan.unknown == ["head repository is unavailable due to GitHub API
error; retrying"]
+ assert plan.new_comments == []
+
+
+def test_deleted_head_repository_retries_on_api_error():
+ plan, _, _ = _classify(_pr(head_repo=None), GREEN,
verify_head_repo_unavailable=lambda: False)
+
+ assert plan.state is None
+ assert plan.close is False
+ assert plan.unknown == ["head repository is unavailable due to GitHub API
error; retrying"]
+ assert plan.new_comments == []
+
+
+def test_check_head_repo_unavailable():
+ client = mock.Mock()
+ client.rest_get.return_value = {"head": {"repo": None}}
+ assert check_head_repo_unavailable(client, "apache/airflow", 123) is True
+
+ client.rest_get.return_value = {"head": {"repo": {"full_name":
"user/repo"}}}
+ assert check_head_repo_unavailable(client, "apache/airflow", 123) is False
+
+ client.rest_get.return_value = None
+ assert check_head_repo_unavailable(client, "apache/airflow", 123) is False
+
+ client.rest_get.side_effect = GitHubError("500 Internal Server Error")
+ assert check_head_repo_unavailable(client, "apache/airflow", 123) is False
def test_malformed_marker_is_reported_and_rewritten():
@@ -2362,6 +2426,44 @@ def
test_fork_observations_use_the_fork_name_and_latest_suite(client):
assert client.graphql.call_args.args[1] == {"o0": "someone", "n0":
"my-airflow", "s0": "sha1"}
+def test_fetch_fork_observations_accepts_workflow_dispatch(client):
+ pr = parse_pull_request(_pr_node(1, head_repo="someone/my-airflow",
head_sha="sha1"))
+ client.graphql.return_value = {
+ "pr0": {
+ "object": {
+ "checkSuites": {
+ "nodes": [
+ {
+ "createdAt": "2026-11-02T00:00:00Z",
+ "updatedAt": "2026-11-02T00:40:00Z",
+ "status": "COMPLETED",
+ "conclusion": "SUCCESS",
+ "workflowRun": {
+ "databaseId": 88,
+ "runAttempt": 1,
+ "url": "dispatch-run",
+ "event": "workflow_dispatch",
+ "workflow": {"resourcePath": "/x/ci-amd.yml"},
+ },
+ },
+ ]
+ }
+ }
+ }
+ }
+
+ observations = fetch_fork_observations(client, [pr])
+
+ assert observations[1] == ForkObservation(
+ suite_status="COMPLETED",
+ suite_conclusion="SUCCESS",
+ run_url="dispatch-run",
+ run_id=88,
+ run_attempt=1,
+ completed_at=datetime(2026, 11, 2, 0, 40, tzinfo=UTC),
+ )
+
+
def test_state_comment_is_found_beyond_the_first_page(client):
marker = _marker("running")
first_page = {
@@ -3432,6 +3534,34 @@ def test_unfinished_gate_change_closure_is_retried():
assert "Closing this pull request because it changes the fork CI gate." in
plan.new_comments[0]
+def test_author_reopen_allowed_when_gate_change_removed():
+ pr = _pr(
+ marker=_marker("gate_modified", days_ago=1),
+ last_reopened_by_anyone_at=NOW,
+ )
+ plan, _, _ = _classify(pr, GREEN, gate_change=None)
+
+ assert plan.state == ForkState.GREEN
+ assert plan.close is False
+
+
+def test_author_reopen_reclosed_when_gate_change_remains():
+ pr = _pr(
+ marker=_marker("gate_modified", days_ago=1),
+ last_reopened_by_anyone_at=NOW,
+ )
+ plan, _, _ = _classify(
+ pr, GREEN, gate_change="it changes
`dev/breeze/src/airflow_breeze/utils/fork_ci_gate.py`"
+ )
+
+ assert plan.state == ForkState.GATE_MODIFIED
+ assert plan.close is True
+ assert (
+ "changes the fork CI gate: it changes
`dev/breeze/src/airflow_breeze/utils/fork_ci_gate.py`"
+ in plan.new_comments[0]
+ )
+
+
PILOT_BASE = "enabled = true\n[pilot]\nother = 2026-10-01T00:00:00Z\n"
@@ -3526,6 +3656,12 @@ def _gate_change_finder(client, changed, head_files,
base_files=None, head_sha="
"it changes `.github/workflows/fork-ci-monitor.yml`",
id="monitor-moved-away",
),
+ pytest.param(
+ [{"filename":
"dev/breeze/src/airflow_breeze/utils/fork_ci_reconcile.py"}],
+ {},
+ "it changes
`dev/breeze/src/airflow_breeze/utils/fork_ci_reconcile.py`",
+ id="reconciler-code-protected",
+ ),
pytest.param(
[{"filename": ".github/fork-ci-gate.toml"}],
{".github/fork-ci-gate.toml": "enabled = false\n"},
@@ -3934,6 +4070,150 @@ def test_mark_older_pilot_pull_request(
assert (marked.ci_run_at, marked.pilot_joined_at) == (expected_ci_run_at,
expected_pilot_joined_at)
+def test_gate_protected_paths_includes_all_five_files():
+ expected = {
+ "dev/breeze/src/airflow_breeze/utils/fork_ci_gate.py",
+ "dev/breeze/src/airflow_breeze/utils/fork_ci_github.py",
+ "dev/breeze/src/airflow_breeze/utils/fork_ci_messages.py",
+ "dev/breeze/src/airflow_breeze/utils/fork_ci_reconcile.py",
+ ".github/workflows/fork-ci-monitor.yml",
+ }
+ assert expected == GATE_PROTECTED_PATHS
+
+
[email protected](
+ ("login", "expected"),
+ [
+ ("github-actions", True),
+ ("github-actions[bot]", True),
+ ("airflow-triage-bot", True),
+ ("Airflow-Triage-Bot", True),
+ ("codecov", True),
+ ("codecov[bot]", True),
+ ("Codecov[Bot]", True),
+ ("dependabot", True),
+ ("dependabot[bot]", True),
+ ("asf-ci[bot]", True),
+ ("asfbot", True),
+ ("ASFBot", True),
+ ("custom-linter[bot]", True),
+ ("ayushmeel", False),
+ ("potiuk", False),
+ ("", False),
+ (None, False),
+ ],
+)
+def test_is_bot(login, expected):
+ assert _is_bot(login) is expected
+
+
+def test_setup_required_inactivity_clock_resets_on_new_commit_sha():
+ pr = _pr(
+ marker=_marker("setup_required", days_ago=7, sha="old_sha"),
+ head_sha="new_sha",
+ )
+ plan, _, _ = _classify(pr, ForkObservation(workflow_state="disabled_fork"))
+
+ assert plan.close is False
+ assert plan.state == ForkState.SETUP_REQUIRED
+
+
+def test_setup_required_inactivity_clock_resets_on_recent_human_comment():
+ pr = _pr(
+ marker=_marker("setup_required", days_ago=7),
+ last_comment_at=NOW - timedelta(days=2),
+ )
+ plan, _, _ = _classify(pr, ForkObservation(workflow_state="disabled_fork"))
+
+ assert plan.close is False
+ assert plan.state == ForkState.SETUP_REQUIRED
+
+
+def test_parse_pull_request_tracks_reviews_and_comments():
+ node = _pr_node(
+ 1,
+ latestReviews={
+ "nodes": [
+ {
+ "createdAt": "2026-11-05T00:00:00Z",
+ "author": {"login": "reviewer"},
+ "comments": {
+ "nodes": [
+ {
+ "createdAt": "2026-11-06T12:00:00Z",
+ "author": {"login": "code-suggester"},
+ },
+ {
+ "createdAt": "2026-11-09T00:00:00Z",
+ "author": {"login": "codecov[bot]"},
+ },
+ ]
+ },
+ }
+ ]
+ },
+ latestComments={
+ "nodes": [
+ {
+ "createdAt": "2026-11-07T00:00:00Z",
+ "author": {"login": "human-author"},
+ },
+ {
+ "createdAt": "2026-11-08T00:00:00Z",
+ "author": {"login": "Airflow-Triage-Bot"},
+ },
+ ]
+ },
+ )
+ pr = parse_pull_request(node)
+ assert pr.last_comment_at == datetime(2026, 11, 7, 0, 0, tzinfo=UTC)
+
+
+def test_fetch_fork_observations_orders_by_created_at_and_run_attempt(client):
+ pr = parse_pull_request(_pr_node(1, head_repo="someone/my-airflow",
head_sha="sha1"))
+ client.graphql.return_value = {
+ "pr0": {
+ "object": {
+ "checkSuites": {
+ "nodes": [
+ {
+ "createdAt": "2026-11-02T00:00:00Z",
+ "updatedAt": "2026-11-02T00:30:00Z",
+ "status": "COMPLETED",
+ "conclusion": "FAILURE",
+ "workflowRun": {
+ "databaseId": 88,
+ "runAttempt": 1,
+ "url": "attempt-1",
+ "event": "workflow_dispatch",
+ "workflow": {"resourcePath": "/x/ci-amd.yml"},
+ },
+ },
+ {
+ "createdAt": "2026-11-02T00:00:00Z",
+ "updatedAt": "2026-11-02T00:50:00Z",
+ "status": "COMPLETED",
+ "conclusion": "SUCCESS",
+ "workflowRun": {
+ "databaseId": 88,
+ "runAttempt": 2,
+ "url": "attempt-2",
+ "event": "workflow_dispatch",
+ "workflow": {"resourcePath": "/x/ci-amd.yml"},
+ },
+ },
+ ]
+ }
+ }
+ }
+ }
+
+ observations = fetch_fork_observations(client, [pr])
+ assert observations[1].run_attempt == 2
+ assert observations[1].suite_conclusion == "SUCCESS"
+ assert observations[1].run_url == "attempt-2"
+
+
def
test_older_pilot_pull_request_stays_checked_for_tampering_after_its_gated_run_is_deleted():
config = dataclasses.replace(CONFIG, enabled=False, pilot={"someone":
JOINED})