This is an automated email from the ASF dual-hosted git repository.
yuqi1129 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gravitino.git
The following commit(s) were added to refs/heads/main by this push:
new 8b2b30dbd2 [#14151] improvement(ci): Stop Fork CI on the first failed
job and report results faster (#14152)
8b2b30dbd2 is described below
commit 8b2b30dbd29c5e903d64d153205a4bc03d3f1062
Author: Qi Yu <[email protected]>
AuthorDate: Fri Oct 23 09:59:13 2026 +0800
[#14151] improvement(ci): Stop Fork CI on the first failed job and report
results faster (#14152)
### What changes were proposed in this pull request?
- `fork-ci.yml`: add a `Stop on first failure` job that polls the run's
jobs once a minute, logs progress (`19/28 jobs completed, 1 failed;
waiting for ...`), and cancels the run as soon as any job fails or times
out. It is the only job with `actions: write`, runs only trusted script
code (no PR build or tests), and uses `ubuntu-latest` because
`ubuntu-slim` stops jobs after 15 minutes.
- `fork-ci-status.cjs`: a run that ended because a job failed is
reported as `failure` with the failed job in the check title (e.g.
`Failed: backend / BackendIT ...`), instead of `cancelled`.
- `update-fork-ci.cjs`: scale the scheduled sweep budget with the hourly
quota (`limit / 24`, at least 60 calls), and keep a scheduled sweep
polling for 11 minutes, one pass every two minutes (repositories with
small quotas keep a single pass). Fork runs cannot notify upstream, and
GitHub fires the five-minute schedule only every 10-20 minutes on
apache/gravitino, so results used to appear 15-40 minutes after the fork
run finished.
- `asf-allowlist-trusted.yml`, `conflict-marker-check.yml`: run on
`ubuntu-slim`. On the shared `ubuntu-latest` pool these few-second jobs
queued for more than three hours, leaving the required **ASF Allowlist
Check** pending. The allowlist action installs its own Python and
supports `ubuntu-slim`.
- `fork-ci-status.cjs` / `update-fork-ci.cjs`: when the run listing
returns no fork run, show a waiting check (`Waiting for Fork CI`)
instead of `action_required` "Enable Actions in the fork", and never
replace a run already recorded on the check with an empty listing.
During the current GitHub Actions incident, `listWorkflowRuns`
intermittently returned 0 runs for existing runs, which turned reopened
PRs into "Enable Actions".
- CONTRIBUTING: describe the new re-run behavior, and that closing and
reopening a PR refreshes the check but cannot re-run CI in the fork.
### Why are the changes needed?
GitHub only allows "Re-run failed jobs" after the whole run completes,
and Fork CI runs all suites in one run, so a single early failure meant
waiting about two hours for the slowest suite before retrying. The PR
check also stayed "in progress" for that time. With 60 calls per sweep,
a finished fork run could take 30+ minutes to appear on apache PRs.
Fix: #14151
### Does this PR introduce _any_ user-facing change?
Contributors: one failed job now stops the rest of the Fork CI run;
"Re-run failed jobs" is available immediately and also re-runs the jobs
that were stopped.
### How was this patch tested?
- 41 Node tests in `.github/scripts/tests/fork-ci.test.cjs` (new: cancel
on first failure, watcher ignores itself and exits after all jobs
complete, failure title, single `actions: write` job on `ubuntu-latest`,
quota-scaled sweep budget, waiting on a missing run, keeping a recorded
run, multi-pass sweeps that mirror a run finishing within the window);
actionlint.
- Sandbox (a fork standing in for apache/gravitino, contributor fork
with a forced allowlist failure and a long stand-in job): the failure
cancelled the run 12 seconds later, the upstream check showed `failure |
Failed: asf_allowlist / asf-allowlist-check`, and "Re-run failed jobs"
was available immediately and re-ran the cancelled jobs (Required CI
succeeded). A run with an 18-minute job passed with the watcher exiting
on its own, and a real test failure in `build / changes` cancelled a run
within one poll, with the progress lines in the watcher log.
- Sandbox polling sweep (5 passes in one job, 00:41-00:51): a fork run
that failed at 00:43:09 showed `Fork CI=failure` at 00:43:43, and one
that passed at 00:49:14 showed `success` at 00:50:40, with no other
event. On `ubuntu-slim`, Conflict Marker Check and ASF Allowlist Check
reported 40-70 seconds after the push.
- Sandbox: after close/reopen, a PR that had been wrongly marked "Enable
Actions" showed `Waiting for Fork CI` and then the fork run once it was
listed again.
- This PR changes `.github/workflows`, so its own Fork CI check needs a
trusted run dispatched by a committer.
---
.github/scripts/fail-fast.cjs | 44 +++++++++
.github/scripts/fork-ci-status.cjs | 18 +++-
.github/scripts/tests/fork-ci.test.cjs | 143 ++++++++++++++++++++++++++--
.github/scripts/update-fork-ci.cjs | 125 +++++++++++++++---------
.github/workflows/asf-allowlist-trusted.yml | 4 +-
.github/workflows/conflict-marker-check.yml | 3 +-
.github/workflows/fork-ci-status-update.yml | 3 +-
.github/workflows/fork-ci.yml | 20 ++++
CONTRIBUTING.md | 2 +-
9 files changed, 297 insertions(+), 65 deletions(-)
diff --git a/.github/scripts/fail-fast.cjs b/.github/scripts/fail-fast.cjs
new file mode 100644
index 0000000000..c3c1c55b10
--- /dev/null
+++ b/.github/scripts/fail-fast.cjs
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+'use strict';
+const STOPPED = ['failure', 'timed_out'];
+
+// Jobs of one run can only be re-run after the whole run completes, and the
slowest suite takes
+// hours. Cancel the run as soon as any job fails, so the author can re-run it
right away.
+async function watch({ github, context, core, self, interval = 60000, sleep =
ms => new Promise(resolve => setTimeout(resolve, ms)) }) {
+ const attempt = Number(process.env.GITHUB_RUN_ATTEMPT || 1);
+ let idle = 0;
+ // Dependent jobs appear only once their needs finish, so require two quiet
polls before exiting.
+ while (idle < 2) {
+ const jobs = (await
github.paginate(github.rest.actions.listJobsForWorkflowRunAttempt, {
+ ...context.repo, run_id: context.runId, attempt_number: attempt,
per_page: 100
+ })).filter(job => job.name !== self);
+ const failed = jobs.filter(job =>
STOPPED.includes(job.conclusion)).map(job => job.name);
+ const running = jobs.filter(job => job.status !== 'completed').map(job =>
job.name);
+ core.info(`${jobs.length - running.length}/${jobs.length} jobs completed,
` +
+ `${failed.length} failed` + (running.length ? `; waiting for
${running.slice(0, 5).join(', ')}` +
+ (running.length > 5 ? ` and ${running.length - 5} more` : '') : ''));
+ if (failed.length) {
+ core.error(`Cancelling the run because these jobs failed:
${failed.join(', ')}. Re-run failed jobs once it is cancelled.`);
+ await github.rest.actions.cancelWorkflowRun({ ...context.repo, run_id:
context.runId });
+ return failed;
+ }
+ idle = running.length ? 0 : idle + 1;
+ if (idle < 2) await sleep(interval);
+ }
+ core.info('All jobs completed without a failure.');
+ return [];
+}
+module.exports = { watch };
diff --git a/.github/scripts/fork-ci-status.cjs
b/.github/scripts/fork-ci-status.cjs
index e3508f4c69..c7fb9af971 100644
--- a/.github/scripts/fork-ci-status.cjs
+++ b/.github/scripts/fork-ci-status.cjs
@@ -24,9 +24,15 @@ const requiredAction = note => ({ status: 'completed',
conclusion: 'action_requi
function evaluate(pr, run, jobs, baseSha = pr.base.sha) {
const required = jobs.find(job => job.name === requiredName(pr, baseSha));
if (run.status !== 'completed') return { status: 'in_progress', title: 'Fork
CI is running' };
- if (run.conclusion !== 'success') return { status: 'completed',
- conclusion: ['failure', 'cancelled', 'timed_out',
'action_required'].includes(run.conclusion)
- ? run.conclusion : 'action_required', title: `Fork CI ${run.conclusion}`
};
+ if (run.conclusion !== 'success') {
+ // Fork CI cancels its own run on the first failed job, so report that
job, not the cancellation.
+ const failed = jobs.filter(job => ['failure',
'timed_out'].includes(job.conclusion)).map(job => job.name);
+ if (failed.length) return { status: 'completed', conclusion: 'failure',
+ title: `Failed: ${failed[0]}${failed.length > 1 ? ` (+${failed.length -
1} more)` : ''}`.slice(0, 200) };
+ return { status: 'completed',
+ conclusion: ['failure', 'cancelled', 'timed_out',
'action_required'].includes(run.conclusion)
+ ? run.conclusion : 'action_required', title: `Fork CI
${run.conclusion}` };
+ }
if (!required || required.conclusion !== 'success') {
return requiredAction('The run did not finish Required CI for this exact
head, base branch and base SHA. ' +
'Update the branch and push again, or ask a committer to dispatch a
trusted Fork CI run.');
@@ -115,7 +121,11 @@ function createService(github) {
const trusted = await findRun(pr, true);
let selected = trusted && (!ordinary || (trusted.run.updated_at ||
trusted.run.created_at) >= (ordinary.run.updated_at || ordinary.run.created_at)
|| !sameDefinitions)
? trusted : ordinary;
- if (!selected) return { result: requiredAction('Enable Actions in the fork
and push the branch, or dispatch trusted Fork CI.') };
+ // Run listings can lag behind, e.g. right after a push or during GitHub
incidents, so keep
+ // waiting instead of asking the author to act.
+ if (!selected) return { result: { status: 'queued', title: 'Waiting for
Fork CI',
+ note: 'No Fork CI run was found for this commit yet. If none starts,
enable Actions in your fork and push again, ' +
+ 'or ask a committer to dispatch trusted Fork CI.' } };
const { owner, repo, run } = selected;
if (!selected.trusted && !sameDefinitions) return { ...selected, fork:
ordinary?.run,
result: requiredAction('CI workflow/action/script definitions or branch
protection differ from the base. ' +
diff --git a/.github/scripts/tests/fork-ci.test.cjs
b/.github/scripts/tests/fork-ci.test.cjs
index f494acec49..7d922fbd87 100644
--- a/.github/scripts/tests/fork-ci.test.cjs
+++ b/.github/scripts/tests/fork-ci.test.cjs
@@ -251,7 +251,7 @@ async function withPublisher(fn) {
}
test('publisher binds the check to the exact base and current run attempt',
async () => withPublisher(async ({ ctx, stateFile, core }) => {
const { github, writes } = publisherMock();
- await update({ github, context: ctx, core, stateFile });
+ await update({ github, context: ctx, core, stateFile, sleep: async () => {}
});
assert.equal(writes.length, 1);
const check = writes[0].create;
assert.equal(check.conclusion, 'success');assert.equal(check.head_sha, A);
@@ -261,7 +261,7 @@ test('publisher binds the check to the exact base and
current run attempt', asyn
test('a push or retarget during publication cannot publish stale success',
async () => withPublisher(async ({ ctx, stateFile, core }) => {
const changed = pr(); changed.base.sha = C;
const { github, writes } = publisherMock({ current: changed });
- await update({ github, context: ctx, core, stateFile });
+ await update({ github, context: ctx, core, stateFile, sleep: async () => {}
});
assert.equal(writes.length, 0);
}));
test('manual publication refreshes a cached final success and observes a
failed rerun', async () => withPublisher(async ({ ctx, stateFile, core }) => {
@@ -271,13 +271,13 @@ test('manual publication refreshes a cached final success
and observes a failed
await fs.writeFile(stateFile, JSON.stringify({ cursor: 0, prs: { 7: { sha:
A, base: B, base_ref: 'main',
updated_at: pr().updated_at, status: 'completed', checked_at: new
Date().toISOString(), completed_at: new Date().toISOString() } } }));
const { github, writes, counters } = publisherMock({ latest: old, rerun: {
...run(), run_attempt: 2, conclusion: 'failure' } });
- await update({ github, context: ctx, core, stateFile });
+ await update({ github, context: ctx, core, stateFile, sleep: async () => {}
});
assert.ok(counters.runs > 0); assert.equal(writes[0].create.conclusion,
'failure');
assert.equal(JSON.parse(writes[0].create.output.text).run_attempt, 2);
}));
test('publisher failures remain observable and are retried instead of cached',
async () => withPublisher(async ({ ctx, stateFile, core, failures }) => {
const { github, writes } = publisherMock({ failTrees: true });
- await update({ github, context: ctx, core, stateFile });
+ await update({ github, context: ctx, core, stateFile, sleep: async () => {}
});
assert.equal(writes.length, 0); assert.equal(failures.length, 1);
await assert.rejects(readFile(stateFile, 'utf8'), { code: 'ENOENT' });
}));
@@ -285,7 +285,7 @@ test('PR signals prioritize their head and bypass settled
state', async () => wi
ctx.eventName = 'workflow_run';delete ctx.payload.inputs;
ctx.payload.workflow_run = { name: 'Fork CI PR Signal', head_sha: A, event:
'pull_request' };
const { github, writes } = publisherMock();
- await update({ github, context: ctx, core, stateFile });
+ await update({ github, context: ctx, core, stateFile, sleep: async () => {}
});
assert.equal(writes.length, 1);
}));
@@ -297,7 +297,7 @@ test('scheduled publication ignores poisoned cache state
and recomputes definiti
} } }));
const { github, writes } = publisherMock();
github.rest.git = serviceMock({ changed: true }).rest.git;
- await update({ github, context: ctx, core, stateFile });
+ await update({ github, context: ctx, core, stateFile, sleep: async () => {}
});
assert.equal(writes[0].create.conclusion, 'action_required');
const workflow = await readFile(path.join(__dirname,
'../../workflows/fork-ci-status-update.yml'), 'utf8');
assert.ok(!workflow.includes('actions/cache/'));
@@ -385,7 +385,7 @@ test('scheduled sweeps probe recently updated PRs first and
refresh only running
response.data.workflow_runs = response.data.workflow_runs.map(run => ({
...run, head_sha: args.head_sha }));
return response;
};
- await update({ github, context: ctx, core, stateFile });
+ await update({ github, context: ctx, core, stateFile, sleep: async () => {}
});
// The settled check is probed for its fork run once, which is unchanged, so
it is not refreshed.
const written = writes.map(write => write.create || write.update);
assert.equal(runsFor.filter(sha => sha === settled.head.sha).length, 1);
@@ -395,7 +395,7 @@ test('scheduled sweeps probe recently updated PRs first and
refresh only running
test('a failed fork run is not summarized as verified', async () =>
withPublisher(async ({ ctx, stateFile, core }) => {
const { github, writes } = publisherMock({ rerun: { ...run(), conclusion:
'failure' } });
- await update({ github, context: ctx, core, stateFile });
+ await update({ github, context: ctx, core, stateFile, sleep: async () => {}
});
assert.equal(writes[0].create.conclusion, 'failure');
assert.doesNotMatch(writes[0].create.output.summary, /Verified/);
}));
@@ -414,7 +414,7 @@ test('scheduled sweeps refresh a settled check when its
fork run is rerun', asyn
const listPulls = github.rest.pulls.list, paginate = github.paginate;
github.paginate = async (endpoint, args) => endpoint === listPulls ?
[recent] : paginate(endpoint, args);
github.rest.pulls.get = async () => ({ data: recent });
- await update({ github, context: ctx, core, stateFile });
+ await update({ github, context: ctx, core, stateFile, sleep: async () => {}
});
assert.equal(writes.length, 1);
assert.equal(writes[0].create.conclusion, 'failure');
assert.equal(JSON.parse(writes[0].create.output.text).fork_run_attempt, 2);
@@ -427,7 +427,7 @@ test('a publisher that observed an older attempt does not
overwrite a newer resu
const check = { id: 55, app: { slug: 'github-actions' }, status:
latest.status, conclusion: latest.conclusion,
output: { text: JSON.stringify({ owner: 'fork', repo: 'repo', run_id: 1,
run_attempt: latest.attempt }) } };
const { github, writes } = publisherMock({ latest: check, rerun: {
...run(), ...observed } });
- await update({ github, context: ctx, core, stateFile });
+ await update({ github, context: ctx, core, stateFile, sleep: async () =>
{} });
assert.equal(writes.length, 0);
}
}));
@@ -451,3 +451,126 @@ test('checkout-and-sync does not persist credentials by
default', async () => {
const action = await readFile(path.join(__dirname,
'../../actions/checkout-and-sync/action.yml'), 'utf8');
assert.match(action, /persist-credentials:\n(?:\s+.*\n)*?\s+default:
'false'/);
});
+
+const { watch } = require('../fail-fast.cjs');
+function watcherMock(polls) {
+ const calls = { cancel: 0, polls: 0 };
+ const github = { rest: { actions: { listJobsForWorkflowRunAttempt() {},
+ cancelWorkflowRun: async args => { calls.cancel++; calls.run_id =
args.run_id; } } },
+ paginate: async (endpoint, args) => { calls.attempt = args.attempt_number;
return polls[Math.min(calls.polls++, polls.length - 1)]; } };
+ const errors = [];
+ return { github, calls, errors, context: { repo: { owner: 'fork', repo:
'repo' }, runId: 42 },
+ core: { error: message => errors.push(message), info: message =>
errors.push(message) }, self: 'Stop on first failure', sleep: async () => {} };
+}
+const job = (name, status, conclusion = null) => ({ name, status, conclusion
});
+
+test('the first failed job cancels the rest of the fork run', async () => {
+ const mock = watcherMock([
+ [job('Stop on first failure', 'in_progress'), job('build / build',
'in_progress')],
+ [job('Stop on first failure', 'in_progress'), job('build / build',
'in_progress'),
+ job('backend / BackendIT (h2)', 'completed', 'failure')]]);
+ assert.deepEqual(await watch(mock), ['backend / BackendIT (h2)']);
+ assert.equal(mock.calls.cancel, 1); assert.equal(mock.calls.run_id, 42);
+ // Every poll reports progress, so the watcher is never a black box.
+ assert.match(mock.errors[0], /0\/1 jobs completed, 0 failed; waiting for
build \/ build$/);
+ assert.match(mock.errors.at(-1), /Cancelling the run because these jobs
failed: backend \/ BackendIT \(h2\)/);
+});
+
+test('the watcher ignores itself and exits without cancelling once all jobs
pass', async () => {
+ process.env.GITHUB_RUN_ATTEMPT = '2';
+ try {
+ const done = [job('Stop on first failure', 'completed', 'failure'),
job('build / build', 'completed', 'success'),
+ job('charts / lint-test', 'completed', 'skipped')];
+ const mock = watcherMock([[job('build / build', 'in_progress')], done,
done]);
+ assert.deepEqual(await watch(mock), []);
+ assert.equal(mock.calls.cancel, 0); assert.equal(mock.calls.polls, 3);
assert.equal(mock.calls.attempt, 2);
+ assert.equal(mock.errors.at(-1), 'All jobs completed without a failure.');
+ } finally { delete process.env.GITHUB_RUN_ATTEMPT; }
+});
+
+test('a run cancelled by fail-fast is reported as the failed job', () => {
+ const p = pr();
+ const failed = [...jobs(p), { name: 'backend / BackendIT (h2)', conclusion:
'failure' }, { name: 'spark / SparkIT', conclusion: 'cancelled' }];
+ const result = evaluate(p, { ...run(), conclusion: 'cancelled' }, failed);
+ assert.equal(result.conclusion, 'failure'); assert.equal(result.title,
'Failed: backend / BackendIT (h2)');
+ assert.equal(evaluate(p, { ...run(), conclusion: 'cancelled' },
jobs(p)).conclusion, 'cancelled');
+});
+
+test('only the fail-fast job can cancel the run', async () => {
+ const workflow = await readFile(path.join(__dirname,
'../../workflows/fork-ci.yml'), 'utf8');
+ assert.equal(workflow.match(/actions: write/g).length, 1);
+ assert.match(workflow, /fail_fast:\n(?:.*\n)*?\s+permissions:\n\s+contents:
read\n\s+actions: write\n/);
+ assert.match(workflow, /^permissions:\n\s+contents: read\n\s+actions:
read\n/m);
+ // ubuntu-slim cancels jobs after 15 minutes; the watcher must outlive the
slowest suite.
+ assert.match(workflow, /fail_fast:\n(?:.*\n)*?\s+runs-on: ubuntu-latest\n/);
+});
+
+test('scheduled sweeps scale their budget with the hourly quota', async () =>
withPublisher(async ({ ctx, stateFile, core }) => {
+ ctx.eventName = 'schedule'; delete ctx.payload.inputs;
+ const probed = async limit => {
+ const { github } = publisherMock();
+ const recent = Array.from({ length: 60 }, (_, i) => ({ ...pr(), number:
200 + i, updated_at: new Date().toISOString(),
+ head: { ...pr().head, sha: (200 + i).toString(16).padStart(40, 'f') }
}));
+ const listPulls = github.rest.pulls.list, listForRef =
github.rest.checks.listForRef, paginate = github.paginate;
+ // Count paginated requests, which is what each probe makes.
+ let count = () => {};
+ github.hook = { before(_, hook) { count = hook; } };
+ github.rest.rateLimit.get = async () => ({ data: { resources: { core: {
remaining: limit, limit } } } });
+ const seen = new Set();
+ github.paginate = async (endpoint, args) => {
+ count();
+ if (endpoint === listPulls) return recent;
+ if (endpoint === listForRef) { seen.add(args.ref); return [{ id: 1, app:
{ slug: 'github-actions' }, status: 'completed',
+ conclusion: 'success', output: { text: JSON.stringify({ run_id: 1,
run_attempt: 1 }) } }]; }
+ return paginate(endpoint, args);
+ };
+ // Each pass waits two minutes of fake time, which also advances the
rotation.
+ let t = Date.now(), passes = 0;
+ await update({ github, context: ctx, core, stateFile, clock: () => t,
sleep: async ms => { t += ms; passes++; } });
+ return { seen: seen.size, passes: passes + 1 };
+ };
+ // A small quota keeps a single pass; a large one keeps polling within the
loop window.
+ const small = await probed(1000), large = await probed(15000);
+ assert.equal(small.passes, 1); assert.ok(small.seen < 60);
+ assert.equal(large.passes, 6); assert.equal(large.seen, 60);
+}));
+
+test('a missing fork run waits instead of asking the author to act', async ()
=> {
+ const github = serviceMock();
+ github.rest.actions.listWorkflowRuns = async () => ({ data: { workflow_runs:
[] } });
+ const { result } = await createService(github).mirror(pr());
+ assert.equal(result.status, 'queued'); assert.equal(result.conclusion,
undefined);
+ assert.match(result.note, /enable Actions in your fork/);
+});
+
+test('an incomplete run listing does not replace a recorded run', async () =>
withPublisher(async ({ ctx, stateFile, core }) => {
+ const green = { id: 55, app: { slug: 'github-actions' }, status:
'completed', conclusion: 'success',
+ output: { text: JSON.stringify({ owner: 'fork', repo: 'repo', run_id: 1,
run_attempt: 1 }) } };
+ const { github, writes } = publisherMock({ latest: green });
+ github.rest.actions.listWorkflowRuns = async () => ({ data: { workflow_runs:
[] } });
+ await update({ github, context: ctx, core, stateFile, sleep: async () => {}
});
+ assert.equal(writes.length, 0);
+}));
+
+test('a scheduled sweep mirrors a fork run that finishes during its polling
window', async () => withPublisher(async ({ ctx, stateFile, core }) => {
+ ctx.eventName = 'schedule'; delete ctx.payload.inputs;
+ const recent = { ...pr(), updated_at: new Date().toISOString() };
+ const { github, writes } = publisherMock({ rerun: { ...run(), status:
'in_progress', conclusion: null } });
+ const listPulls = github.rest.pulls.list, paginate = github.paginate, list =
github.rest.actions.listWorkflowRuns;
+ let latest = null;
+ github.paginate = async (endpoint, args) => endpoint === listPulls ? [recent]
+ : endpoint === github.rest.checks.listForRef ? (latest ? [latest] : []) :
paginate(endpoint, args);
+ github.rest.checks.create = async args => { writes.push({ create: args });
latest = { id: writes.length, app: { slug: 'github-actions' }, ...args }; };
+ github.rest.checks.update = async args => { writes.push({ update: args });
Object.assign(latest, args); };
+ github.rest.pulls.get = async () => ({ data: recent });
+ github.rest.rateLimit.get = async () => ({ data: { resources: { core: {
remaining: 15000, limit: 15000 } } } });
+ let t = Date.now(), finished = false;
+ github.rest.actions.listWorkflowRuns = async args => {
+ const response = await list(args);
+ if (finished) response.data.workflow_runs =
response.data.workflow_runs.map(r => ({ ...r, status: 'completed', conclusion:
'success' }));
+ return response;
+ };
+ await update({ github, context: ctx, core, stateFile, clock: () => t, sleep:
async ms => { t += ms; finished = true; } });
+ assert.equal((writes[0].create || writes[0].update).status, 'in_progress');
+ assert.equal((writes.at(-1).create || writes.at(-1).update).conclusion,
'success');
+}));
diff --git a/.github/scripts/update-fork-ci.cjs
b/.github/scripts/update-fork-ci.cjs
index d993e01f35..dabfc130b2 100644
--- a/.github/scripts/update-fork-ci.cjs
+++ b/.github/scripts/update-fork-ci.cjs
@@ -16,8 +16,13 @@
const { createService } = require('./fork-ci-status.cjs');
const RECENT_MS = 6 * 60 * 60 * 1000;
+// Fork runs finish without an upstream event, and GitHub fires five-minute
schedules only every
+// 10-20 minutes on busy repositories. A scheduled sweep therefore keeps
polling for LOOP_MS, one
+// pass every PASS_MS, staying below the 15-minute job limit of ubuntu-slim.
+const LOOP_MS = 11 * 60 * 1000;
+const PASS_MS = 2 * 60 * 1000;
-async function update({ github, context, core }) {
+async function update({ github, context, core, sleep = ms => new
Promise(resolve => setTimeout(resolve, ms)), clock = Date.now }) {
const signal = context.eventName === 'workflow_run' &&
context.payload.workflow_run.name === 'Fork CI PR Signal';
const trustedSignal = context.eventName === 'workflow_run' &&
context.payload.workflow_run.event === 'workflow_dispatch' &&
@@ -25,22 +30,29 @@ async function update({ github, context, core }) {
const manual = context.eventName === 'workflow_dispatch' || !!trustedSignal
|| signal;
const requested = context.payload.inputs?.['pull-request'] || (trustedSignal
&& trustedSignal[1]);
if (requested && !/^[1-9][0-9]*$/.test(requested)) throw new Error('Invalid
PR number.');
- let prs = requested
- ? [(await github.rest.pulls.get({ ...context.repo, pull_number:
Number(requested) })).data]
- : await github.paginate(github.rest.pulls.list, { ...context.repo, state:
'open', per_page: 100 });
- if (signal) prs = prs.filter(pr => pr.head.sha ===
context.payload.workflow_run.head_sha);
- const now = Date.now();
+ async function listPrs() {
+ const prs = requested
+ ? [(await github.rest.pulls.get({ ...context.repo, pull_number:
Number(requested) })).data]
+ : await github.paginate(github.rest.pulls.list, { ...context.repo,
state: 'open', per_page: 100 });
+ return signal ? prs.filter(pr => pr.head.sha ===
context.payload.workflow_run.head_sha) : prs;
+ }
// Never restore decisions or scheduling state from Actions caches: PR code
dispatched
// on the default branch can write those caches using its runner runtime
token.
const service = createService(github);
const availableBases = new Map();
const rate = (await github.rest.rateLimit.get()).data.resources.core;
// Bound work even while settled successes are watched. Rotate the starting
PR to avoid starving
- // older PRs, and leave quota for signal workflows and other repository
automation.
- const budget = manual ? Math.max(0, rate.remaining - 100) : Math.min(60,
Math.max(0, rate.remaining - 100));
+ // older PRs, and leave quota for signal workflows and other repository
automation. Organizations on
+ // GitHub Enterprise Cloud, such as apache, get 15,000 requests per hour:
spend at most about half of
+ // the hourly quota across five-minute sweeps, and never less than 60 calls
per sweep.
+ const perSweep = Math.max(60, Math.floor(rate.limit / 24));
+ const budget = manual ? Math.max(0, rate.remaining - 100) :
Math.min(perSweep, Math.max(0, rate.remaining - 100));
+ // Small quotas keep one pass per sweep; larger ones split the sweep budget
across passes.
+ const passes = manual ? 1 : Math.max(1, Math.min(Math.ceil(LOOP_MS /
PASS_MS), Math.floor(budget / 60)));
+ const slotMs = passes > 1 ? PASS_MS : 300000;
let calls = 0;
github.hook.before('request', () => { calls++; });
- const failures = [];
+ const failures = new Set();
const recordOf = check => { try { return JSON.parse(check?.output?.text ||
'{}'); } catch { return {}; } };
async function latestCheck(pr) {
const checks = await github.paginate(github.rest.checks.listForRef, {
@@ -82,6 +94,8 @@ async function update({ github, context, core }) {
// attempt of a run from the same repository, or a completed result with a
running one.
const latest = await latestCheck(pr);
const oldRecord = recordOf(latest);
+ // A run already recorded for this commit still exists; a listing that
missed it is incomplete.
+ if (!record.run_id && oldRecord.run_id) return;
if (oldRecord.owner === record.owner && oldRecord.repo === record.repo &&
record.run_id &&
(oldRecord.run_id > record.run_id || (oldRecord.run_id ===
record.run_id &&
(oldRecord.run_attempt > record.run_attempt ||
@@ -99,51 +113,68 @@ async function update({ github, context, core }) {
if (pr.state !== 'open') return;
try {
await work();
+ failures.delete(pr.number);
} catch (error) {
- failures.push(pr.number);
+ failures.add(pr.number);
core.warning(`Could not refresh #${pr.number}: ${error.message}`);
}
}
let processed = 0;
- const visited = new Set();
- // Fork runs, including reruns, finish without an upstream event. Such PRs
were updated recently,
- // so probe those first. A completed result changes only with a push or a
trusted run, which both
- // signal a refresh, or with a fork rerun, so refresh it only when the fork
run id or attempt changed.
- const recent = manual ? [] : prs.filter(pr => now -
Date.parse(pr.updated_at) < RECENT_MS)
- .sort((a, b) => Date.parse(b.updated_at) - Date.parse(a.updated_at));
- // Probes take two calls, so rotate them too and keep a third of the budget
for older PRs.
- const recentBudget = Math.floor(budget * 2 / 3);
- const recentStart = !recent.length ? 0 : (Math.floor(now / 300000) *
Math.max(1, Math.floor(recentBudget / 4))) % recent.length;
- let probed = 0;
- for (let offset = 0; offset < recent.length; offset++) {
- if (calls + 15 > recentBudget) break; // One PR may need trees, runs,
paginated jobs and a check write.
- const pr = recent[(recentStart + offset) % recent.length];
- probed++;
- await visit(pr, async () => {
- const latest = await latestCheck(pr);
- if (latest?.status === 'completed') {
- if (!pr.head.repo) return;
- const record = recordOf(latest);
- const fork = (await service.findRun(pr, false))?.run;
- if (!fork || (fork.id === (record.fork_run_id ?? record.run_id) &&
- fork.run_attempt === (record.fork_run_attempt ??
record.run_attempt))) return;
- }
- await refresh(pr);
- });
+ let visited = new Set();
+ let prs = [];
+ async function pass(budget) {
+ const now = clock();
+ prs = await listPrs();
+ processed = 0;
+ visited = new Set();
+ // Fork runs, including reruns, finish without an upstream event. Such PRs
were updated recently,
+ // so probe those first. A completed result changes only with a push or a
trusted run, which both
+ // signal a refresh, or with a fork rerun, so refresh it only when the
fork run id or attempt changed.
+ const recent = manual ? [] : prs.filter(pr => now -
Date.parse(pr.updated_at) < RECENT_MS)
+ .sort((a, b) => Date.parse(b.updated_at) - Date.parse(a.updated_at));
+ // Probes take two calls, so rotate them too and keep a third of the
budget for older PRs.
+ const recentBudget = Math.floor(budget * 2 / 3);
+ const recentStart = !recent.length ? 0 : (Math.floor(now / slotMs) *
Math.max(1, Math.floor(recentBudget / 4))) % recent.length;
+ let probed = 0;
+ for (let offset = 0; offset < recent.length; offset++) {
+ if (calls + 15 > recentBudget) break; // One PR may need trees, runs,
paginated jobs and a check write.
+ const pr = recent[(recentStart + offset) % recent.length];
+ probed++;
+ await visit(pr, async () => {
+ const latest = await latestCheck(pr);
+ if (latest?.status === 'completed') {
+ if (!pr.head.repo) return;
+ const record = recordOf(latest);
+ const fork = (await service.findRun(pr, false))?.run;
+ if (!fork || (fork.id === (record.fork_run_id ?? record.run_id) &&
+ fork.run_attempt === (record.fork_run_attempt ??
record.run_attempt))) return;
+ }
+ await refresh(pr);
+ });
+ }
+ // Rotate the rest using trusted wall-clock time, so old PRs are
eventually visited without
+ // accepting a contributor-writable cursor. Advance by about one sweep's
capacity.
+ const step = Math.max(1, Math.floor(budget / 20));
+ const start = manual || !prs.length ? 0 : (Math.floor(now / slotMs) *
step) % prs.length;
+ for (let offset = 0; offset < prs.length; offset++) {
+ if (calls + 15 > budget) break;
+ const pr = prs[(start + offset) % prs.length];
+ if (visited.has(pr.number)) continue;
+ await visit(pr, () => refresh(pr));
+ }
+ core.info(`Visited ${processed}/${prs.length} PRs (probed
${probed}/${recent.length} recently updated); ` +
+ `${calls}/${budget} API calls; starting index ${start}.`);
}
- // Rotate the rest using trusted wall-clock time, so old PRs are eventually
visited without
- // accepting a contributor-writable cursor. Advance by about one sweep's
capacity.
- const step = Math.max(1, Math.floor(budget / 20));
- const start = manual || !prs.length ? 0 : (Math.floor(now / 300000) * step)
% prs.length;
- for (let offset = 0; offset < prs.length; offset++) {
- if (calls + 15 > budget) break;
- const pr = prs[(start + offset) % prs.length];
- if (visited.has(pr.number)) continue;
- await visit(pr, () => refresh(pr));
+ const deadline = clock() + LOOP_MS;
+ for (let i = 0; i < passes; i++) {
+ if (i) {
+ if (clock() + PASS_MS > deadline) break;
+ await sleep(PASS_MS);
+ calls = 0;
+ }
+ await pass(Math.floor(budget / passes));
}
- core.info(`Visited ${processed}/${prs.length} PRs (probed
${probed}/${recent.length} recently updated); ` +
- `${calls}/${budget} API calls; starting index ${start}.`);
if (manual && processed < prs.length) core.setFailed('API budget exhausted;
dispatch again for a specific PR after the quota resets.');
- if (failures.length) core.setFailed(`Unable to refresh: ${failures.map(n =>
`#${n}`).join(', ')}`);
+ if (failures.size) core.setFailed(`Unable to refresh: ${[...failures].map(n
=> `#${n}`).join(', ')}`);
}
module.exports = { update };
diff --git a/.github/workflows/asf-allowlist-trusted.yml
b/.github/workflows/asf-allowlist-trusted.yml
index 950eaed107..d4c50b6058 100644
--- a/.github/workflows/asf-allowlist-trusted.yml
+++ b/.github/workflows/asf-allowlist-trusted.yml
@@ -33,7 +33,9 @@ jobs:
if: >-
github.repository == 'apache/gravitino' &&
(github.event_name == 'workflow_dispatch' ||
github.event.workflow_run.event == 'pull_request')
- runs-on: ubuntu-latest
+ # A required check: keep it off the shared ubuntu-latest pool, where small
jobs can queue for hours.
+ # The allowlist action installs its own Python, so it supports ubuntu-slim.
+ runs-on: ubuntu-slim
timeout-minutes: 10
permissions:
contents: read
diff --git a/.github/workflows/conflict-marker-check.yml
b/.github/workflows/conflict-marker-check.yml
index 7b14e9cc7b..fa6711bc7f 100644
--- a/.github/workflows/conflict-marker-check.yml
+++ b/.github/workflows/conflict-marker-check.yml
@@ -37,7 +37,8 @@ concurrency:
jobs:
conflict-marker-check:
name: conflict-marker-check
- runs-on: ubuntu-latest
+ # Only git and grep; ubuntu-slim avoids queueing behind integration tests
on ubuntu-latest.
+ runs-on: ubuntu-slim
steps:
- uses: actions/checkout@v4
with:
diff --git a/.github/workflows/fork-ci-status-update.yml
b/.github/workflows/fork-ci-status-update.yml
index 30ee724606..5a421dc010 100644
--- a/.github/workflows/fork-ci-status-update.yml
+++ b/.github/workflows/fork-ci-status-update.yml
@@ -45,7 +45,8 @@ jobs:
github.event.workflow_run.event == 'workflow_dispatch' ||
github.event.workflow_run.event == 'push')
runs-on: ubuntu-slim
- timeout-minutes: 10
+ # Scheduled sweeps poll for 11 minutes; ubuntu-slim stops jobs after 15.
+ timeout-minutes: 14
permissions:
actions: read
checks: write
diff --git a/.github/workflows/fork-ci.yml b/.github/workflows/fork-ci.yml
index 6ec755a8d8..78e3f8fc43 100644
--- a/.github/workflows/fork-ci.yml
+++ b/.github/workflows/fork-ci.yml
@@ -253,6 +253,26 @@ jobs:
sync: ${{ needs.eligibility.outputs.sync }}
uses: ./.github/workflows/asf-allowlist-check.yml
+ fail_fast:
+ name: Stop on first failure
+ needs: eligibility
+ if: needs.eligibility.outputs.valid == 'true'
+ # Not ubuntu-slim: it limits jobs to 15 minutes, and this job lasts as
long as the slowest suite.
+ runs-on: ubuntu-latest
+ timeout-minutes: 360
+ permissions:
+ contents: read
+ actions: write
+ steps:
+ - uses: actions/checkout@v4
+ with:
+ persist-credentials: false
+ - uses: actions/github-script@v7
+ with:
+ retries: 3
+ script: |
+ await require('./.github/scripts/fail-fast.cjs').watch({ github,
context, core, self: 'Stop on first failure' });
+
required_ci:
name: Required CI / ${{ needs.eligibility.outputs.base-ref }} / ${{
needs.eligibility.outputs.base-sha }} / ${{ needs.eligibility.outputs.head-sha
}}
# !cancelled() rather than always(), so a cancelled run does not wait for
a runner here.
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index a2931f2f30..3d33709492 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -99,7 +99,7 @@ Pull request tests run with GitHub Actions in your fork, not
in `apache/gravitin
3. Open the pull request. Its **Fork CI** check verifies the suite jobs and
the exact tested head, base branch and base SHA. A pre-PR run is accepted only
if its inferred base matches the PR. After retargeting a PR, push again or ask
a committer for a trusted run. The separate **ASF Allowlist Check** scans the
PR's workflow YAML using reviewed upstream code.
4. Keep your branch up to date before merging. Required checks use strict
branch protection. If the base changes, update your branch and run CI again.
The **Fork CI** check can stay green after the base branch moves on; strict
branch protection, not the check, blocks the merge until the branch is updated.
-If the check asks you to enable Fork CI, enable Actions in your fork, merge or
rebase your branch onto the latest base branch, and push again. To re-run flaky
tests, re-run the failed jobs in your fork; reviewers cannot re-run them for
you. Please fix failures before requesting a review.
+If the check asks you to enable Fork CI, enable Actions in your fork, merge or
rebase your branch onto the latest base branch, and push again. Fork CI cancels
the rest of the run as soon as one job fails, so you can re-run failed jobs in
your fork right away; this also re-runs the jobs it cancelled. Reviewers cannot
re-run them for you. Closing and reopening a pull request only refreshes the
check from the latest fork run; to run CI again, re-run the workflow in your
fork or push again. [...]
Known limitations of Fork CI:
