This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/master by this push:
new c3526970d562 chore(ci): bridge fork CI coverage to Codecov (#20475)
c3526970d562 is described below
commit c3526970d562365475cdbfc44664a846b7f27549
Author: voonhous <[email protected]>
AuthorDate: Thu Nov 12 09:14:36 2026 +0000
chore(ci): bridge fork CI coverage to Codecov (#20475)
With fork CI, a fork PR's tests run on the fork, which cannot use
apache/hudi's Codecov token and is not linked to the PR, so fork PRs
lose the Codecov comment during review.
- New upload-coverage action replaces the 12 Codecov steps. In
apache/hudi it uploads as before; on a fork it saves
jacoco-report.xml as an artifact named by commit, job and flag.
- New fork_coverage_bridge.yml, every 15 minutes: matches each open
fork PR's head commit to the fork's finished Java CI and Java CI
Engines runs, downloads their coverage artifacts with its own
GITHUB_TOKEN, uploads them to Codecov for that PR and commit under
the same flags, and records a "Coverage (fork)" check so each pair
of runs uploads once. Each PR is recorded from its own upload jobs,
so a failed upload only makes the next sweep retry its own PR.
- Artifacts are untrusted data: names must match a strict pattern and
a flag allowlist, sizes are capped, only the one XML member is
extracted, and no fork code is checked out or executed.
- Codecov maps JaCoCo package paths through the uploader's file list,
so the bridge loads the PR head's file list with a tree-only fetch
from apache/hudi.
- 21 node tests of the selection and recording scripts run in
validate-pr-and-source.
Part of #20471.
---
.github/actions/upload-coverage/action.yml | 61 ++++
.github/ci/scripts/test-fork-coverage-bridge.cjs | 421 +++++++++++++++++++++++
.github/workflows/README.md | 5 +-
.github/workflows/bot.yml | 36 +-
.github/workflows/fork_coverage_bridge.yml | 354 +++++++++++++++++++
.github/workflows/java_ci_engines.yml | 12 +-
.github/workflows/validate_source.yml | 16 +-
7 files changed, 859 insertions(+), 46 deletions(-)
diff --git a/.github/actions/upload-coverage/action.yml
b/.github/actions/upload-coverage/action.yml
new file mode 100644
index 000000000000..ea0032bf7c21
--- /dev/null
+++ b/.github/actions/upload-coverage/action.yml
@@ -0,0 +1,61 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: Upload coverage
+description: >-
+ Uploads ./jacoco-report.xml to Codecov in apache/hudi, or as a run artifact
on a fork for
+ fork_coverage_bridge.yml to upload.
+
+# A composite action cannot read secrets, so the caller passes the Codecov
token in.
+inputs:
+ flags:
+ description: Codecov flag of the report.
+ required: true
+ token:
+ description: Codecov token of apache/hudi. Empty on forks.
+ required: true
+
+runs:
+ using: composite
+ steps:
+ # Status functions in a composite action see only the action's own steps,
so these steps run
+ # even when the job has failed and the caller's step was started by
always().
+ - name: Upload coverage to Codecov
+ if: github.repository == 'apache/hudi'
+ uses: codecov/codecov-action@75cd11691c0faa626561e295848008c8a7dddffe #
v5
+ with:
+ files: ./jacoco-report.xml
+ disable_search: true
+ flags: ${{ inputs.flags }}
+ token: ${{ inputs.token }}
+ # A fork run cannot use apache/hudi's Codecov token, and Codecov does not
link a run on the
+ # fork to the apache/hudi PR. fork_coverage_bridge.yml downloads this
artifact and uploads it
+ # for the PR instead. It parses the name: `__` separates the parts, since
job ids and flags
+ # both contain hyphens. Several jobs share a flag, so the job id keeps the
name unique; a job
+ # whose matrix has several uploading entries would also need a matrix
value in the name.
+ # A re-run of a job keeps the run id, and upload-artifact refuses a second
artifact of the same
+ # name in one run, so without overwrite a re-run of a failed fork job
would fail here even when
+ # its tests pass. The replacement gets a new artifact id, which the bridge
picks as the newest.
+ - name: Upload coverage as an artifact for the coverage bridge
+ if: github.repository != 'apache/hudi'
+ uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a #
v7.0.1
+ with:
+ name: coverage-${{ github.sha }}__${{ github.job }}__${{ inputs.flags
}}
+ path: jacoco-report.xml
+ retention-days: 7
+ if-no-files-found: ignore
+ overwrite: true
diff --git a/.github/ci/scripts/test-fork-coverage-bridge.cjs
b/.github/ci/scripts/test-fork-coverage-bridge.cjs
new file mode 100644
index 000000000000..7f83fde46954
--- /dev/null
+++ b/.github/ci/scripts/test-fork-coverage-bridge.cjs
@@ -0,0 +1,421 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// Tests the github-script steps of fork_coverage_bridge.yml against mocked
GitHub APIs, in the
+// style of test-fork-ci-status.cjs.
+const assert = require('node:assert/strict');
+const { execFileSync } = require('node:child_process');
+const { readFileSync } = require('node:fs');
+const path = require('node:path');
+const { test } = require('node:test');
+
+const bridgeWorkflow = JSON.parse(execFileSync('python3', [
+ '-c', 'import json, sys, yaml; print(json.dumps(yaml.safe_load(sys.stdin)))',
+], {
+ input: readFileSync(path.join(__dirname,
'../../workflows/fork_coverage_bridge.yml')),
+ encoding: 'utf8',
+}));
+const AsyncFunction = Object.getPrototypeOf(async function () {}).constructor;
+const select = new AsyncFunction('github', 'context', 'core', 'Date',
+ bridgeWorkflow.jobs.select.steps[0].with.script);
+const record = new AsyncFunction('github', 'context', 'core', 'process',
+ bridgeWorkflow.jobs.record.steps[0].with.script);
+const now = Date.UTC(2026, 9, 20, 12);
+class Clock extends Date { static now() { return now; } }
+const MB = 1024 * 1024;
+
+function makePr(number, { sha = String(number % 10).repeat(40), owner =
'contributor', ref = 'feature',
+ base = 'master', updated = now, full_name } = {}) {
+ return {
+ number, state: 'open', updated_at: new Date(updated).toISOString(),
+ head: { sha, ref, repo: { owner: { login: owner }, name: 'hudi',
full_name: full_name || owner + '/hudi' } },
+ base: { ref: base },
+ };
+}
+const pr = makePr(10);
+const sha = pr.head.sha;
+const run = (id = 1, extra = {}) => ({
+ id, run_attempt: 1, event: 'push', head_sha: sha, head_branch: 'feature',
status: 'completed', conclusion: 'success',
+ ...extra,
+});
+let nextArtifactId = 1000;
+const artifact = (name, extra = {}) => ({ id: nextArtifactId++, name, expired:
false, size_in_bytes: MB, ...extra });
+const coverage = (job, flag, headSha = sha) => 'coverage-' + headSha + '__' +
job + '__' + flag;
+// The runs of a fork, keyed by workflow file. By default bot.yml has run 1
and java_ci_engines.yml run 2.
+const workflows = (bot = [run(1)], engines = [run(2)]) => ({ 'bot.yml': bot,
'java_ci_engines.yml': engines });
+const WORKFLOW_IDS = ['bot.yml', 'java_ci_engines.yml'];
+const runIds = (bot, engines, botAttempt = 1, enginesAttempt = 1) => [
+ { workflow: 'bot.yml', run_id: bot, run_attempt: botAttempt },
+ { workflow: 'java_ci_engines.yml', run_id: engines, run_attempt:
enginesAttempt },
+];
+
+// runs are keyed by fork owner and workflow file, artifacts by run id; a
missing key answers 404.
+async function executeSelect({ pulls = [pr], runs = { contributor: workflows()
}, artifacts, checks = [] } = {}) {
+ artifacts = artifacts || {
+ 1: [artifact(coverage('test-utilities', 'utilities'))],
+ 2: [artifact(coverage('test-flink-1', 'flink-integration-tests'))],
+ };
+ const outputs = {}, warnings = [], calls = [];
+ const github = { rest: { checks: {}, actions: {}, pulls: {} } };
+ const notFound = () => Object.assign(new Error('Not Found'), { status: 404
});
+ github.rest.pulls.list = async () => {};
+ github.rest.checks.listForRef = async () => {};
+ github.rest.actions.listWorkflowRunArtifacts = async () => {};
+ github.rest.actions.listWorkflowRuns = async (params) => {
+ calls.push(['runs', params]);
+ assert.ok(WORKFLOW_IDS.includes(params.workflow_id), params.workflow_id);
+ if (!(params.owner in runs)) throw notFound();
+ const fork = runs[params.owner];
+ if (!(params.workflow_id in fork)) throw notFound();
+ return { data: { workflow_runs: fork[params.workflow_id] } };
+ };
+ github.paginate = async (fn, params) => {
+ if (fn === github.rest.checks.listForRef) {
+ assert.equal(params.owner, 'apache');
+ assert.equal(params.check_name, 'Coverage (fork)');
+ return checks.filter((item) => item.ref === params.ref);
+ }
+ assert.equal(fn, github.rest.actions.listWorkflowRunArtifacts);
+ calls.push(['artifacts', params]);
+ if (!(params.run_id in artifacts)) throw notFound();
+ return artifacts[params.run_id];
+ };
+ github.paginate.iterator = async function* (fn, params) {
+ assert.equal(fn, github.rest.pulls.list);
+ assert.equal(params.base, 'master');
+ assert.equal(params.sort, 'updated');
+ for (let i = 0; i < pulls.length; i += 100) yield { data: pulls.slice(i, i
+ 100) };
+ };
+ await select(github, { repo: { owner: 'apache', repo: 'hudi' } }, {
+ info: () => {}, warning: (message) => warnings.push(message),
+ setOutput: (name, value) => { outputs[name] = JSON.parse(value); },
+ }, Clock);
+ return { ...outputs, warnings, calls };
+}
+
+test('selects the artifacts of both completed fork runs under their own run
ids', async () => {
+ const { artifacts, prs, calls } = await executeSelect();
+ const entry = { pr: 10, sha, branch: 'contributor:feature', fork:
'contributor/hudi' };
+ assert.deepEqual(artifacts, [
+ { ...entry, workflow: 'bot.yml', run_id: 1, artifact_id:
artifacts[0].artifact_id, flags: 'utilities' },
+ { ...entry, workflow: 'java_ci_engines.yml', run_id: 2, artifact_id:
artifacts[1].artifact_id,
+ flags: 'flink-integration-tests' },
+ ]);
+ assert.deepEqual(prs, [{ pr: 10, sha, runs: runIds(1, 2) }]);
+ assert.deepEqual(calls.map(([kind, params]) => [kind, params.workflow_id ||
params.run_id]), [
+ ['runs', 'bot.yml'], ['runs', 'java_ci_engines.yml'], ['artifacts', 1],
['artifacts', 2],
+ ]);
+});
+test('same-repository PRs and PRs to other branches are ignored', async () => {
+ const result = await executeSelect({ pulls: [
+ makePr(10, { owner: 'apache', full_name: 'Apache/Hudi' }),
+ makePr(10, { base: 'release-1.2' }),
+ ] });
+ assert.deepEqual(result.artifacts, []);
+ assert.deepEqual(result.prs, []);
+ assert.deepEqual(result.calls, []);
+});
+test('only completed push and manual runs of the current head are used', async
() => {
+ const { artifacts, prs } = await executeSelect({
+ runs: { contributor: workflows([
+ run(9, { status: 'in_progress', conclusion: null }),
+ run(8, { event: 'pull_request' }),
+ run(7, { head_branch: 'other' }),
+ run(6, { event: 'workflow_dispatch' }),
+ run(5),
+ ]) },
+ artifacts: { 6: [artifact(coverage('test-utilities', 'utilities'))] },
+ });
+ assert.deepEqual(artifacts.map((item) => item.run_id), [6]);
+ assert.deepEqual(prs, [{ pr: 10, sha, runs: runIds(6, 2) }]);
+});
+test('a PR whose head moved past either fork run is skipped', async () => {
+ const moved = { head_sha: 'f'.repeat(40) };
+ for (const runs of [workflows([run(1, moved)]), workflows(undefined, [run(2,
moved)])]) {
+ const result = await executeSelect({ runs: { contributor: runs } });
+ assert.deepEqual(result.artifacts, []);
+ assert.deepEqual(result.prs, []);
+ assert.equal(result.calls.filter(([kind]) => kind === 'artifacts').length,
0);
+ }
+});
+test('a PR is skipped while either workflow has no completed run', async () =>
{
+ const running = { status: 'in_progress', conclusion: null };
+ for (const runs of [workflows(undefined, [run(2, running)]),
workflows([run(1, running)])]) {
+ const result = await executeSelect({ runs: { contributor: runs } });
+ assert.deepEqual(result.artifacts, []);
+ assert.deepEqual(result.prs, []);
+ assert.equal(result.calls.filter(([kind]) => kind === 'artifacts').length,
0);
+ assert.deepEqual(result.warnings, []);
+ }
+});
+test('a fork without one of the workflows is skipped without a warning', async
() => {
+ for (const runs of [{ 'bot.yml': [run(1)] }, { 'java_ci_engines.yml':
[run(2)] }]) {
+ const result = await executeSelect({ runs: { contributor: runs } });
+ assert.deepEqual(result.artifacts, []);
+ assert.deepEqual(result.prs, []);
+ assert.equal(result.calls.filter(([kind]) => kind === 'artifacts').length,
0);
+ assert.deepEqual(result.warnings, []);
+ }
+});
+test('a PR recorded with both current run ids is skipped, a foreign check is
not', async () => {
+ const recorded = { ref: sha, app: { slug: 'github-actions' }, external_id:
'hudi-fork-coverage:10:1.1:2.1' };
+ const skipped = await executeSelect({ checks: [recorded] });
+ assert.deepEqual(skipped.artifacts, []);
+ assert.deepEqual(skipped.prs, []);
+ assert.equal(skipped.calls.filter(([kind]) => kind === 'artifacts').length,
0);
+ for (const check of [
+ { ...recorded, app: { slug: 'other-app' } },
+ { ...recorded, external_id: 'hudi-fork-coverage:20:1.1:2.1' },
+ { ...recorded, external_id: 'hudi-fork-coverage:10:1:2' },
+ ]) {
+ assert.equal((await executeSelect({ checks: [check] })).artifacts.length,
2);
+ }
+});
+test('a new run of either workflow changes the id, so the PR is uploaded
again', async () => {
+ const recorded = { ref: sha, app: { slug: 'github-actions' }, external_id:
'hudi-fork-coverage:10:1.1:2.1' };
+ const artifacts = {
+ 1: [artifact(coverage('test-utilities', 'utilities'))],
+ 2: [artifact(coverage('test-flink-1', 'flink-integration-tests'))],
+ 3: [artifact(coverage('test-utilities', 'utilities'))],
+ 4: [artifact(coverage('test-flink-1', 'flink-integration-tests'))],
+ };
+ for (const [runs, ids] of [
+ [workflows([run(3, { event: 'workflow_dispatch' }), run(1)]), [3, 2]],
+ [workflows(undefined, [run(4, { event: 'workflow_dispatch' }), run(2)]),
[1, 4]],
+ ]) {
+ const result = await executeSelect({ runs: { contributor: runs },
artifacts, checks: [recorded] });
+ assert.deepEqual(result.prs, [{ pr: 10, sha, runs: runIds(...ids) }]);
+ assert.deepEqual(new Set(result.artifacts.map((item) => item.run_id)), new
Set(ids));
+ }
+});
+test('a re-run of either workflow keeps the run id but is uploaded again',
async () => {
+ const recorded = { ref: sha, app: { slug: 'github-actions' }, external_id:
'hudi-fork-coverage:10:1.2:2.1' };
+ const name = coverage('test-utilities', 'utilities');
+ // The re-run job overwrote its artifact, which got a new id.
+ const rerun = artifact(name);
+ const artifacts = { 1: [rerun], 2: [artifact(coverage('test-flink-1',
'flink-integration-tests'))] };
+ const same = await executeSelect({
+ runs: { contributor: workflows([run(1, { run_attempt: 2 })]) }, artifacts,
checks: [recorded],
+ });
+ assert.deepEqual(same.prs, []);
+ assert.equal(same.calls.filter(([kind]) => kind === 'artifacts').length, 0);
+ for (const [runs, attempts] of [
+ [workflows([run(1, { run_attempt: 3 })]), [3, 1]],
+ [workflows([run(1, { run_attempt: 2 })], [run(2, { run_attempt: 2 })]),
[2, 2]],
+ ]) {
+ const result = await executeSelect({ runs: { contributor: runs },
artifacts, checks: [recorded] });
+ assert.deepEqual(result.prs, [{ pr: 10, sha, runs: runIds(1, 2,
...attempts) }]);
+ assert.equal(result.artifacts.find((item) => item.run_id ===
1).artifact_id, rerun.id);
+ }
+ const { writes } = await executeRecord({
+ prs: [{ pr: 10, sha, runs: runIds(1, 2, 3, 1) }],
+ artifacts: [{ pr: 10, fork: 'contributor/hudi', artifact_id: rerun.id,
flags: 'utilities' }],
+ });
+ assert.deepEqual(writes.map((item) => item.external_id),
['hudi-fork-coverage:10:1.3:2.1']);
+});
+test('names, SHAs, flags, expired and oversized artifacts are validated',
async () => {
+ const valid = [
+ artifact(coverage('test-flink-1', 'flink-integration-tests')),
+ artifact(coverage('test-spark-java17-java-tests-part1',
'spark-java-tests'), { size_in_bytes: 50 * MB }),
+ ];
+ const { artifacts } = await executeSelect({ artifacts: { 1: [
+ ...valid,
+ artifact('coverage-' + sha + '-test-utilities-utilities'),
+ artifact(coverage('test utilities', 'utilities')),
+ artifact(coverage('test-utilities', 'Utilities')),
+ artifact(coverage('test-utilities', 'utilities') + '.xml'),
+ artifact('x' + coverage('test-utilities', 'utilities')),
+ artifact(coverage('test-utilities', 'utilities', 'b'.repeat(40))),
+ artifact(coverage('test-utilities', 'made-up-flag')),
+ artifact(coverage('test-common', 'common-and-other-modules'), { expired:
true }),
+ artifact(coverage('test-hadoop', 'hadoop-mr-java-client'), {
size_in_bytes: 50 * MB + 1 }),
+ artifact(coverage('test-scala', 'spark-scala-tests'), { size_in_bytes:
undefined }),
+ artifact('jacoco'),
+ ] } });
+ assert.deepEqual(artifacts.map((item) => item.artifact_id), valid.map((item)
=> item.id));
+ assert.deepEqual(artifacts.map((item) => item.flags),
['flink-integration-tests', 'spark-java-tests']);
+});
+test('a rerun job uploads the same name again and only the newest is kept',
async () => {
+ const name = coverage('test-utilities', 'utilities');
+ const [older, newer] = [artifact(name), artifact(name)];
+ const { artifacts } = await executeSelect({ artifacts: { 1: [newer, older] }
});
+ assert.deepEqual(artifacts.map((item) => item.artifact_id), [newer.id]);
+});
+test('PRs without matching artifacts are not listed', async () => {
+ for (const artifacts of [{ 1: [], 2: [] }, { 1: [artifact('logs')] }]) {
+ const result = await executeSelect({ artifacts });
+ assert.deepEqual(result.artifacts, []);
+ assert.deepEqual(result.prs, []);
+ }
+});
+test('the sweep stops at PRs not updated in the last 24 hours', async () => {
+ const old = makePr(11, { updated: now - 25 * 3600 * 1000 });
+ const result = await executeSelect({ pulls: [old, pr] });
+ assert.deepEqual(result.prs, []);
+ assert.deepEqual(result.calls, []);
+});
+test('forks without the workflow, runs or artifacts are tolerated', async ()
=> {
+ const gone = makePr(11, { owner: 'gone' });
+ const empty = makePr(12, { owner: 'empty', sha: 'c'.repeat(40) });
+ const result = await executeSelect({
+ pulls: [gone, empty, pr],
+ runs: {
+ contributor: workflows(),
+ empty: workflows([run(3, { head_sha: empty.head.sha })], [run(4, {
head_sha: empty.head.sha })]),
+ },
+ });
+ assert.deepEqual(result.prs, [{ pr: 10, sha, runs: runIds(1, 2) }]);
+ assert.deepEqual(result.warnings, []);
+});
+test('other API errors skip only the failing PR and warn', async () => {
+ const broken = makePr(11, { owner: 'broken' });
+ const result = await executeSelect({ pulls: [broken, pr], runs: {
+ contributor: workflows(), get broken() { throw Object.assign(new
Error('API failure'), { status: 500 }); },
+ } });
+ assert.deepEqual(result.prs, [{ pr: 10, sha, runs: runIds(1, 2) }]);
+ assert.deepEqual(result.warnings, ['#11: API failure']);
+});
+test('a sweep uploads at most 60 reports, and a PR cut by the cap is not
recorded', async () => {
+ const pulls = [], runs = {}, artifacts = {};
+ for (let i = 0; i < 7; i++) {
+ const owner = 'contributor' + i;
+ const item = makePr(100 + i, { owner, sha: String(i).repeat(40) });
+ pulls.push(item);
+ // As in Java CI, bot.yml uploads 9 of the 12 reports and
java_ci_engines.yml the other 3.
+ const head = { head_sha: item.head.sha };
+ runs[owner] = workflows([run(10 * i + 1, head)], [run(10 * i + 2, head)]);
+ artifacts[10 * i + 1] = Array.from({ length: 9 }, (_, job) =>
+ artifact(coverage('job-' + job, 'utilities', item.head.sha)));
+ artifacts[10 * i + 2] = Array.from({ length: 3 }, (_, job) =>
+ artifact(coverage('job-' + (9 + job), 'integration-tests',
item.head.sha)));
+ }
+ const result = await executeSelect({ pulls, runs, artifacts });
+ assert.equal(result.artifacts.length, 60);
+ assert.deepEqual(result.prs.map((item) => item.pr), [100, 101, 102, 103,
104]);
+ // The fifth PR filled the cap exactly, so no later PR is looked up.
+ assert.equal(result.calls.filter(([kind, params]) => kind === 'runs' &&
params.owner === 'contributor5').length, 0);
+
+ artifacts[11].push(artifact(coverage('job-12', 'utilities',
pulls[1].head.sha)));
+ const cut = await executeSelect({ pulls, runs, artifacts });
+ assert.equal(cut.artifacts.length, 60);
+ assert.deepEqual(cut.prs.map((item) => item.pr), [100, 101, 102, 103]);
+ assert.equal(cut.artifacts.filter((item) => item.pr === 104).length, 11);
+});
+
+// Every upload job succeeds unless its artifact id is in failedUploads.
+async function executeRecord({ prs, artifacts, failing = [], failedUploads =
[] }) {
+ const writes = [], failures = [], warnings = [], jobCalls = [];
+ const jobs = [{ name: 'Select the fork runs to upload', conclusion:
'success' }, ...artifacts.map((item) => ({
+ name: 'Upload #' + item.pr + ' ' + item.flags + ' (' + item.artifact_id +
')',
+ conclusion: failedUploads.includes(item.artifact_id) ? 'failure' :
'success',
+ }))];
+ const github = {
+ paginate: async (method, params) => (await method(params)).data,
+ rest: {
+ actions: { listJobsForWorkflowRun: async (params) => {
jobCalls.push(params); return { data: jobs }; } },
+ checks: {
+ create: async (payload) => {
+ if (failing.includes(payload.head_sha)) throw new Error('API
failure');
+ writes.push(payload);
+ },
+ },
+ },
+ };
+ await record(github, { repo: { owner: 'apache', repo: 'hudi' }, runId: 77 },
{
+ info: () => {}, warning: (message) => warnings.push(message), setFailed:
(message) => failures.push(message),
+ }, { env: { PRS: JSON.stringify(prs), ARTIFACTS: JSON.stringify(artifacts) }
});
+ return { writes, failures, warnings, jobCalls };
+}
+
+test('record creates one completed check per PR', async () => {
+ const other = makePr(11, { owner: 'other', sha: 'c'.repeat(40) });
+ const { artifacts, prs } = await executeSelect({
+ pulls: [pr, other],
+ runs: {
+ contributor: workflows(),
+ other: workflows([run(3, { head_sha: other.head.sha })], [run(4, {
head_sha: other.head.sha })]),
+ },
+ artifacts: {
+ 1: [artifact(coverage('test-utilities', 'utilities'))],
+ 2: [artifact(coverage('test-flink-1', 'flink-integration-tests'))],
+ 3: [artifact(coverage('test-utilities', 'utilities', other.head.sha))],
+ },
+ });
+ const { writes, failures } = await executeRecord({ prs, artifacts });
+ assert.deepEqual(failures, []);
+ assert.equal(writes.length, 2);
+ assert.deepEqual(writes.map((item) => [item.head_sha, item.external_id]), [
+ [sha, 'hudi-fork-coverage:10:1.1:2.1'], [other.head.sha,
'hudi-fork-coverage:11:3.1:4.1'],
+ ]);
+ for (const item of writes) {
+ assert.equal(item.owner, 'apache');
+ assert.equal(item.name, 'Coverage (fork)');
+ assert.equal(item.status, 'completed');
+ assert.equal(item.conclusion, 'success');
+ assert.match(item.output.summary, /self-reported/);
+ assert.match(item.output.summary, /merge queue/);
+ }
+ assert.equal(writes[0].details_url,
'https://github.com/contributor/hudi/actions/runs/1');
+ assert.ok(writes[0].output.summary.includes(
+ '\n- [bot.yml run 1, attempt
1](https://github.com/contributor/hudi/actions/runs/1)' +
+ '\n- [java_ci_engines.yml run 2, attempt
1](https://github.com/contributor/hudi/actions/runs/2)\n'));
+ assert.ok(writes[1].output.summary.includes(
+ '\n- [bot.yml run 3, attempt
1](https://github.com/other/hudi/actions/runs/3)' +
+ '\n- [java_ci_engines.yml run 4, attempt
1](https://github.com/other/hudi/actions/runs/4)\n'));
+});
+test('record reports a failed check but still records the other PRs', async ()
=> {
+ const prs = [{ pr: 10, sha, runs: runIds(1, 2) }, { pr: 11, sha:
'c'.repeat(40), runs: runIds(3, 4) }];
+ const artifacts = [
+ { pr: 10, fork: 'contributor/hudi', artifact_id: 101, flags: 'utilities' },
+ { pr: 11, fork: 'other/hudi', artifact_id: 111, flags: 'utilities' },
+ ];
+ const { writes, failures } = await executeRecord({ prs, artifacts, failing:
[sha] });
+ assert.deepEqual(writes.map((item) => item.external_id),
['hudi-fork-coverage:11:3.1:4.1']);
+ assert.deepEqual(failures, ['#10: API failure']);
+});
+test('a failed upload leaves only its own PR unrecorded', async () => {
+ const prs = [{ pr: 10, sha, runs: runIds(1, 2) }, { pr: 11, sha:
'c'.repeat(40), runs: runIds(3, 4) }];
+ const artifacts = [
+ { pr: 10, fork: 'contributor/hudi', artifact_id: 101, flags: 'utilities' },
+ { pr: 10, fork: 'contributor/hudi', artifact_id: 102, flags:
'flink-integration-tests' },
+ { pr: 11, fork: 'other/hudi', artifact_id: 111, flags: 'utilities' },
+ ];
+ const { writes, failures, warnings, jobCalls } = await executeRecord({ prs,
artifacts, failedUploads: [102] });
+ assert.deepEqual(writes.map((item) => item.external_id),
['hudi-fork-coverage:11:3.1:4.1']);
+ assert.deepEqual(failures, []);
+ assert.deepEqual(warnings, ['#10: an upload failed, so the next sweep
uploads this PR again']);
+ assert.deepEqual(jobCalls, [{ owner: 'apache', repo: 'hudi', run_id: 77,
filter: 'latest', per_page: 100 }]);
+});
+test('record runs after failed uploads, but not after a failed or empty
select', () => {
+ const { record: job } = bridgeWorkflow.jobs;
+ assert.equal(job.if, "needs.select.result == 'success' &&
needs.select.outputs.prs != '[]' && !cancelled()");
+ assert.deepEqual(job.permissions, { actions: 'read', checks: 'write' });
+});
+
+test('jobs never check out code and pass matrix values only through env', ()
=> {
+ const { jobs } = bridgeWorkflow;
+ for (const job of Object.values(jobs)) {
+ for (const step of job.steps) {
+ assert.doesNotMatch(step.uses || '', /actions\/checkout/);
+ assert.doesNotMatch(step.run || '', /\$\{\{/);
+ }
+ }
+ assert.equal(jobs.select.if, "github.repository == 'apache/hudi'");
+ assert.deepEqual(bridgeWorkflow.permissions, {});
+ assert.equal(jobs.upload.strategy['fail-fast'], false);
+ assert.equal(jobs.upload.strategy['max-parallel'], 4);
+});
diff --git a/.github/workflows/README.md b/.github/workflows/README.md
index ad9d1fc3962b..6025162a8710 100644
--- a/.github/workflows/README.md
+++ b/.github/workflows/README.md
@@ -6,7 +6,8 @@ When updating the pr template, you must consider if updates
need to be made to s
- bot.yml: Java CI. Spark lanes (JDK 11 on Spark 3.5, JDK 17 on Spark 4.2).
Runs on PRs, merge-queue groups and branch pushes; only push runs are
path-filtered. Forks run it on every branch push; upstream it skips every job
on pushes to branches other than master, release-* and branch-0.x, and every
job but the aggregate on PRs from forks to master (see [Pull requests from
forks](#pull-requests-from-forks)). No job depends on another, except the final
`Java CI result` job, which aggregat [...]
- java_ci_engines.yml: Java CI Engines. Flink, bundle validation and
integration tests. Same triggers, fork routing and layout rules as bot.yml; its
aggregate job is `Java CI Engines result`. The two are separate workflows
because the ASF GitHub Actions policy caps the jobs running at once in one
workflow at 20, and a single workflow for both would exceed it
- fork_ci_status.yml: Fork CI status. Mirrors the fork's bot.yml and
java_ci_engines.yml runs for the head commit of each open fork PR to master
onto the PR as one informational `CI (fork)` check, which reports the worse of
the two results. Runs when an upstream Java CI or Java CI Engines run completes
and every 15 minutes for late runs and reruns. It never checks out PR code. Its
tests, .github/ci/scripts/test-fork-ci-status.cjs, also cover the routing and
concurrency of bot.yml and jav [...]
-- validate_source.yml: PR and Source Validation. One job,
`validate-pr-and-source`, the only required status check: it checks the PR
description with scripts/pr_compliance.py, runs that script's own tests and the
fork_ci_status.yml tests, validates the PR title, rejects commits with a Claude
co-author trailer, and runs the source release checks (RAT, binary files,
copyright). Never path-filtered, because a required check must report on every
PR. Also runs on merge-queue groups, where onl [...]
+- fork_coverage_bridge.yml: Fork coverage bridge. Every 15 minutes, uploads to
Codecov the JaCoCo reports that the fork's bot.yml and java_ci_engines.yml runs
stored as artifacts (through .github/actions/upload-coverage) for the head
commit of each recently updated fork PR to master, and records each uploaded
commit as a `Coverage (fork)` check. It never checks out or runs PR code; its
tests are .github/ci/scripts/test-fork-coverage-bridge.cjs
+- validate_source.yml: PR and Source Validation. One job,
`validate-pr-and-source`, the only required status check: it checks the PR
description with scripts/pr_compliance.py, runs that script's own tests and the
fork_ci_status.yml and fork_coverage_bridge.yml tests, validates the PR title,
rejects commits with a Claude co-author trailer, and runs the source release
checks (RAT, binary files, copyright). Never path-filtered, because a required
check must report on every PR. Also runs on [...]
## Pull requests from forks
@@ -16,6 +17,6 @@ To reduce load on ASF's shared runners, pull requests from
forks to `master` run
2. Sync your fork and rebase your branch onto upstream `master` to include the
current CI workflows.
3. Push your branch.
-Each push runs two workflows on your fork, `Java CI` and `Java CI Engines`,
and both appear in your fork's Actions tab. View results and rerun failed jobs
there; the PR's `CI (fork)` check combines both runs and may lag. A push that
changes only docs or images creates no fork run, because both workflows
path-filter push events.
+Each push runs two workflows on your fork, `Java CI` and `Java CI Engines`,
and both appear in your fork's Actions tab. View results and rerun failed jobs
there; the PR's `CI (fork)` check combines both runs and may lag. A push that
changes only docs or images creates no fork run, because both workflows
path-filter push events. The PR's Codecov report arrives up to about 15 minutes
after the fork runs finish, through the coverage bridge
(fork_coverage_bridge.yml).
The `CI (fork)` check links to the fork runs and is informational. When your
PR is approved and added to the merge queue, both workflows run using ASF's
runners against the proposed merge with `master`. Release-branch PRs, and PRs
from branches pushed to apache/hudi itself, keep upstream testing.
diff --git a/.github/workflows/bot.yml b/.github/workflows/bot.yml
index 8383f28e25cc..afc02b803a89 100644
--- a/.github/workflows/bot.yml
+++ b/.github/workflows/bot.yml
@@ -159,10 +159,8 @@ jobs:
run: ./scripts/jacoco/generate_merged_coverage_report.sh
$GITHUB_WORKSPACE
- name: Upload coverage to Codecov
if: always() && github.event_name != 'merge_group'
- uses: codecov/codecov-action@75cd11691c0faa626561e295848008c8a7dddffe
# v5
+ uses: ./.github/actions/upload-coverage
with:
- files: ./jacoco-report.xml
- disable_search: true
flags: spark-client-hadoop-common
token: ${{ secrets.CODECOV_TOKEN }}
@@ -212,10 +210,8 @@ jobs:
run: ./scripts/jacoco/generate_merged_coverage_report.sh
$GITHUB_WORKSPACE
- name: Upload coverage to Codecov
if: always() && github.event_name != 'merge_group'
- uses: codecov/codecov-action@75cd11691c0faa626561e295848008c8a7dddffe
# v5
+ uses: ./.github/actions/upload-coverage
with:
- files: ./jacoco-report.xml
- disable_search: true
flags: utilities
token: ${{ secrets.CODECOV_TOKEN }}
@@ -304,10 +300,8 @@ jobs:
run: ./scripts/jacoco/generate_merged_coverage_report.sh
$GITHUB_WORKSPACE
- name: Upload coverage to Codecov
if: always() && github.event_name != 'merge_group'
- uses: codecov/codecov-action@75cd11691c0faa626561e295848008c8a7dddffe
# v5
+ uses: ./.github/actions/upload-coverage
with:
- files: ./jacoco-report.xml
- disable_search: true
flags: common-and-other-modules
token: ${{ secrets.CODECOV_TOKEN }}
@@ -649,10 +643,8 @@ jobs:
run: ./scripts/jacoco/generate_merged_coverage_report.sh
$GITHUB_WORKSPACE
- name: Upload coverage to Codecov
if: always() && github.event_name != 'merge_group'
- uses: codecov/codecov-action@75cd11691c0faa626561e295848008c8a7dddffe
# v5
+ uses: ./.github/actions/upload-coverage
with:
- files: ./jacoco-report.xml
- disable_search: true
flags: hadoop-mr-java-client
token: ${{ secrets.CODECOV_TOKEN }}
@@ -707,10 +699,8 @@ jobs:
run: ./scripts/jacoco/generate_merged_coverage_report.sh
$GITHUB_WORKSPACE
- name: Upload coverage to Codecov
if: always() && github.event_name != 'merge_group'
- uses: codecov/codecov-action@75cd11691c0faa626561e295848008c8a7dddffe
# v5
+ uses: ./.github/actions/upload-coverage
with:
- files: ./jacoco-report.xml
- disable_search: true
flags: spark-java-tests
token: ${{ secrets.CODECOV_TOKEN }}
@@ -775,10 +765,8 @@ jobs:
run: ./scripts/jacoco/generate_merged_coverage_report.sh
$GITHUB_WORKSPACE
- name: Upload coverage to Codecov
if: always() && github.event_name != 'merge_group'
- uses: codecov/codecov-action@75cd11691c0faa626561e295848008c8a7dddffe
# v5
+ uses: ./.github/actions/upload-coverage
with:
- files: ./jacoco-report.xml
- disable_search: true
flags: spark-java-tests
token: ${{ secrets.CODECOV_TOKEN }}
@@ -837,10 +825,8 @@ jobs:
run: ./scripts/jacoco/generate_merged_coverage_report.sh
$GITHUB_WORKSPACE
- name: Upload coverage to Codecov
if: always() && github.event_name != 'merge_group'
- uses: codecov/codecov-action@75cd11691c0faa626561e295848008c8a7dddffe
# v5
+ uses: ./.github/actions/upload-coverage
with:
- files: ./jacoco-report.xml
- disable_search: true
flags: spark-java-tests
token: ${{ secrets.CODECOV_TOKEN }}
@@ -899,10 +885,8 @@ jobs:
run: ./scripts/jacoco/generate_merged_coverage_report.sh
$GITHUB_WORKSPACE
- name: Upload coverage to Codecov
if: always() && github.event_name != 'merge_group'
- uses: codecov/codecov-action@75cd11691c0faa626561e295848008c8a7dddffe
# v5
+ uses: ./.github/actions/upload-coverage
with:
- files: ./jacoco-report.xml
- disable_search: true
flags: spark-scala-tests
token: ${{ secrets.CODECOV_TOKEN }}
@@ -961,10 +945,8 @@ jobs:
run: ./scripts/jacoco/generate_merged_coverage_report.sh
$GITHUB_WORKSPACE
- name: Upload coverage to Codecov
if: always() && github.event_name != 'merge_group'
- uses: codecov/codecov-action@75cd11691c0faa626561e295848008c8a7dddffe
# v5
+ uses: ./.github/actions/upload-coverage
with:
- files: ./jacoco-report.xml
- disable_search: true
flags: spark-scala-tests
token: ${{ secrets.CODECOV_TOKEN }}
diff --git a/.github/workflows/fork_coverage_bridge.yml
b/.github/workflows/fork_coverage_bridge.yml
new file mode 100644
index 000000000000..d17805c59c1f
--- /dev/null
+++ b/.github/workflows/fork_coverage_bridge.yml
@@ -0,0 +1,354 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Uploads the coverage of fork CI runs to Codecov for their apache/hudi PRs. A
fork PR to master
+# runs the Java CI suites on the contributor's fork (see fork_ci_status.yml).
A run there cannot
+# use apache/hudi's Codecov token, and Codecov would not link it to the PR
anyway, so
+# .github/actions/upload-coverage stores each JaCoCo report as a run artifact
named
+# coverage-<sha>__<job id>__<flag> instead. Java CI runs on the fork as two
workflows, bot.yml and
+# java_ci_engines.yml, and both store reports. Every 15 minutes, `select`
finds the newest completed
+# fork run of each for the head commit of each recently updated fork PR and,
once both have
+# completed, lists their coverage artifacts, `upload` uploads each one to
Codecov for the PR and its
+# head commit, and `record` marks the pair of runs as done with a `Coverage
(fork)` check, so later
+# sweeps skip it.
+#
+# This workflow polls because workflow_run in apache/hudi does not fire for
runs in a fork.
+#
+# Artifacts are untrusted data from the contributor. Their names are validated
against a regex
+# and the flag allowlist, their sizes are capped, only the one XML member is
extracted, and
+# nothing from the fork is checked out or executed: of the PR head, `upload`
reads only the list
+# of file paths, which Codecov needs to place the report. A fork can still
forge its own coverage
+# numbers. That is acceptable because coverage does not gate merging.
+# Test with Python 3.11+, PyYAML, and Node.js 24:
+# node --test .github/ci/scripts/test-fork-coverage-bridge.cjs
+name: Fork coverage bridge
+
+on:
+ # Offset from fork_ci_status.yml, so the two sweeps do not hit the API at
the same minute.
+ schedule:
+ - cron: '12,27,42,57 * * * *'
+ workflow_dispatch:
+
+permissions: {}
+
+# A sweep that overlaps the previous one would upload the same reports twice.
Waiting is cheap,
+# and cancelling a sweep would drop uploads that are already under way.
+concurrency:
+ group: fork-coverage-bridge
+ cancel-in-progress: false
+
+jobs:
+ select:
+ if: github.repository == 'apache/hudi'
+ name: Select fork coverage artifacts
+ runs-on: ubuntu-slim
+ timeout-minutes: 10
+ permissions:
+ actions: read
+ checks: read
+ pull-requests: read
+ outputs:
+ artifacts: ${{ steps.select.outputs.artifacts }}
+ prs: ${{ steps.select.outputs.prs }}
+ steps:
+ - name: List the coverage artifacts of recent fork runs
+ id: select
+ uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3
# v9.0.0
+ with:
+ script: |
+ const CHECK = 'Coverage (fork)';
+ // Java CI is split across two workflows to stay under the ASF
limit on jobs running at
+ // once per workflow, and each fork run stores the reports of its
own jobs.
+ const WORKFLOWS = ['bot.yml', 'java_ci_engines.yml'];
+ // The flags of the upload steps in bot.yml and
java_ci_engines.yml. A fork cannot add
+ // flags of its own to apache/hudi's Codecov reports.
+ const FLAGS = new Set([
+ 'common-and-other-modules', 'flink-integration-tests',
'hadoop-mr-java-client',
+ 'integration-tests', 'spark-client-hadoop-common',
'spark-java-tests',
+ 'spark-scala-tests', 'utilities',
+ ]);
+ // The two fork runs of a PR upload 12 reports together, so a
sweep covers at least 5
+ // PRs. PRs beyond the cap wait for the next sweep.
+ const MAX_ARTIFACTS = 60;
+ const MAX_BYTES = 50 * 1024 * 1024;
+ const upstream = { owner: context.repo.owner, repo:
context.repo.repo };
+ const repository = upstream.owner + '/' + upstream.repo;
+ const listing = { ...upstream, state: 'open', base: 'master',
sort: 'updated', direction: 'desc' };
+ const isForkPr = (pr) => pr.state === 'open' && pr.head.repo &&
+ pr.head.repo.full_name.toLowerCase() !==
repository.toLowerCase() && pr.base.ref === 'master';
+ // A re-run of a job keeps the run id and bumps the run attempt.
The id holds both for
+ // each workflow, so a new run or a re-run of either one is
uploaded again.
+ const externalId = (pr, runs) => 'hudi-fork-coverage:' + pr.number
+ ':' +
+ runs.map((run) => run.id + '.' + run.run_attempt).join(':');
+
+ // An absent workflow, run or artifact list is expected on a fork
with Actions
+ // disabled or without one of the workflows. Other API errors must
not pass as missing
+ // coverage.
+ async function optional(request) {
+ try {
+ return await request();
+ } catch (error) {
+ if (error.status === 404) return null;
+ throw error;
+ }
+ }
+
+ // Returns the runs and the valid coverage artifacts of a PR, or
null if there is
+ // nothing to upload.
+ async function collect(pr) {
+ const fork = { owner: pr.head.repo.owner.login, repo:
pr.head.repo.name };
+ const sha = pr.head.sha;
+ const runs = [];
+ for (const workflow of WORKFLOWS) {
+ const data = await optional(() =>
github.rest.actions.listWorkflowRuns({
+ ...fork, workflow_id: workflow, head_sha: sha, branch:
pr.head.ref, per_page: 100,
+ }));
+ // As in fork_ci_status.yml, only push and manual runs test
the branch head. An
+ // unfinished run has not uploaded all of its reports yet. The
PR waits until both
+ // workflows have a completed run, so that its check covers
all of Java CI.
+ const run = (data?.data.workflow_runs || []).filter((item) =>
+ item.status === 'completed' && item.head_sha === sha &&
item.head_branch === pr.head.ref &&
+ ['push', 'workflow_dispatch'].includes(item.event)
+ ).sort((a, b) => b.id - a.id)[0];
+ if (!run) return null;
+ runs.push({ ...run, workflow });
+ }
+
+ const checks = await
github.paginate(github.rest.checks.listForRef, {
+ ...upstream, ref: sha, check_name: CHECK, per_page: 100,
+ });
+ if (checks.some((item) =>
+ item.app?.slug === 'github-actions' && item.external_id ===
externalId(pr, runs))) return null;
+
+ const pattern = new RegExp('^coverage-' + sha +
'__([A-Za-z0-9_-]+)__([a-z0-9-]+)$');
+ const artifacts = [];
+ for (const run of runs) {
+ const listed = await optional(() =>
github.paginate(github.rest.actions.listWorkflowRunArtifacts, {
+ ...fork, run_id: run.id, per_page: 100,
+ })) || [];
+ // A rerun job uploads its artifact again under the same name.
Keep the newest.
+ const newest = new Map();
+ for (const artifact of listed) {
+ const match = pattern.exec(artifact.name);
+ if (!match || !FLAGS.has(match[2]) || artifact.expired ||
+ !(artifact.size_in_bytes <= MAX_BYTES)) continue;
+ if (!(newest.get(artifact.name)?.artifact_id > artifact.id))
{
+ newest.set(artifact.name, {
+ pr: pr.number, sha, branch: fork.owner + ':' +
pr.head.ref, fork: pr.head.repo.full_name,
+ workflow: run.workflow, run_id: run.id, artifact_id:
artifact.id, flags: match[2],
+ });
+ }
+ }
+ artifacts.push(...newest.values());
+ }
+ if (!artifacts.length) return null;
+ return { runs, artifacts: artifacts.sort((a, b) => a.artifact_id
- b.artifact_id) };
+ }
+
+ const artifacts = [];
+ const prs = [];
+ const cutoff = Date.now() - 24 * 3600 * 1000;
+ const pages = github.paginate.iterator(github.rest.pulls.list, {
...listing, per_page: 100 });
+ sweep: for await (const page of pages) {
+ for (const pr of page.data) {
+ if (new Date(pr.updated_at).getTime() < cutoff) break sweep;
+ if (!isForkPr(pr)) continue;
+ let found;
+ // A failing fork must not hold back the coverage of the other
PRs. Its PR is
+ // retried by the next sweep.
+ try {
+ found = await collect(pr);
+ } catch (error) {
+ core.warning('#' + pr.number + ': ' + error.message);
+ continue;
+ }
+ if (!found) continue;
+ const room = MAX_ARTIFACTS - artifacts.length;
+ artifacts.push(...found.artifacts.slice(0, room));
+ // A PR cut by the cap is not recorded, so the next sweep
uploads all of its reports.
+ if (found.artifacts.length <= room) {
+ prs.push({
+ pr: pr.number, sha: pr.head.sha,
+ runs: found.runs.map((run) =>
+ ({ workflow: run.workflow, run_id: run.id, run_attempt:
run.run_attempt })),
+ });
+ }
+ core.info('#' + pr.number + ': ' + found.artifacts.length + '
reports from runs ' +
+ found.runs.map((run) => run.id + '.' +
run.run_attempt).join(' and '));
+ if (artifacts.length >= MAX_ARTIFACTS) break sweep;
+ }
+ }
+ core.setOutput('artifacts', JSON.stringify(artifacts));
+ core.setOutput('prs', JSON.stringify(prs));
+
+ upload:
+ needs: select
+ if: needs.select.outputs.artifacts != '[]'
+ name: 'Upload #${{ matrix.artifact.pr }} ${{ matrix.artifact.flags }} (${{
matrix.artifact.artifact_id }})'
+ runs-on: ubuntu-slim
+ timeout-minutes: 10
+ permissions:
+ actions: read
+ strategy:
+ fail-fast: false
+ max-parallel: 4
+ matrix:
+ artifact: ${{ fromJSON(needs.select.outputs.artifacts) }}
+ # bash, unlike the default shell, sets pipefail, so a failed unzip fails
the extraction.
+ defaults:
+ run:
+ shell: bash
+ # Nothing from the fork is checked out or executed. Matrix values reach
the shell through env
+ # only, never through ${{ }} in a script.
+ steps:
+ # The download endpoint needs a token even for a public repository's
artifacts, and accepts
+ # the GITHUB_TOKEN of any workflow. It redirects to a storage URL on
another host, to which
+ # curl does not forward the Authorization header.
+ - name: Download the artifact
+ env:
+ GITHUB_TOKEN: ${{ github.token }}
+ FORK: ${{ matrix.artifact.fork }}
+ ARTIFACT_ID: ${{ matrix.artifact.artifact_id }}
+ run: |
+ curl -sS -L --fail --max-filesize 52428800 --proto '=https'
--proto-redir '=https' \
+ -H "Authorization: Bearer $GITHUB_TOKEN" \
+ -H 'Accept: application/vnd.github+json' \
+ -o "$RUNNER_TEMP/coverage.zip" \
+
"https://api.github.com/repos/$FORK/actions/artifacts/$ARTIFACT_ID/zip"
+ # The archive is untrusted. It must hold exactly one member,
jacoco-report.xml, of at most
+ # 200 MB. The listed size is only declared by the archive, so the
extraction is capped too.
+ # unzip -p writes the member to stdout, so no path from the archive is
created.
+ - name: Extract the report
+ env:
+ MAX_BYTES: 209715200
+ run: |
+ zip="$RUNNER_TEMP/coverage.zip"
+ report="$RUNNER_TEMP/coverage/jacoco-report.xml"
+ names=$(unzip -Z1 "$zip")
+ read -r size count _ < <(unzip -l "$zip" | tail -n 1)
+ if [ "$names" != 'jacoco-report.xml' ] || [ "$count" != 1 ] || [
"$size" -gt "$MAX_BYTES" ]; then
+ echo "::error::Expected only jacoco-report.xml of at most
$MAX_BYTES bytes in the artifact."
+ exit 1
+ fi
+ mkdir -p "$RUNNER_TEMP/coverage"
+ unzip -p "$zip" jacoco-report.xml | head -c "$((MAX_BYTES + 1))" >
"$report"
+ if [ "$(wc -c < "$report")" -gt "$MAX_BYTES" ]; then
+ echo "::error::jacoco-report.xml exceeds $MAX_BYTES bytes."
+ exit 1
+ fi
+ case "$(head -c 7 "$report")" in
+ '<?xml'* | '<report') ;;
+ *)
+ echo '::error::jacoco-report.xml is not an XML report.'
+ exit 1
+ ;;
+ esac
+ # A JaCoCo report names files by package, and Codecov maps them to
repository paths through
+ # the file list that its uploader reads with git ls-files. Fetch only
the commit and trees
+ # of the PR head from apache/hudi, which keeps every PR head, and load
them into the index:
+ # no file content is downloaded, and nothing is written to the work
tree. The fetch records
+ # apache/hudi as the only remote, so the uploader still reports the
upload for apache/hudi.
+ - name: Load the file list of the PR head
+ env:
+ SHA: ${{ matrix.artifact.sha }}
+ run: |
+ git init -q "$GITHUB_WORKSPACE"
+ git -C "$GITHUB_WORKSPACE" fetch -q --depth=1 --filter=blob:none
--no-tags \
+ "$GITHUB_SERVER_URL/$GITHUB_REPOSITORY.git" "$SHA"
+ git -C "$GITHUB_WORKSPACE" read-tree "$SHA"
+ # File fixes would read source files, which are absent here. They apply
to none of the
+ # languages of Hudi.
+ - name: Upload coverage to Codecov
+ uses: codecov/codecov-action@75cd11691c0faa626561e295848008c8a7dddffe
# v5
+ with:
+ files: ${{ runner.temp }}/coverage/jacoco-report.xml
+ disable_search: true
+ disable_file_fixes: true
+ flags: ${{ matrix.artifact.flags }}
+ override_commit: ${{ matrix.artifact.sha }}
+ override_pr: ${{ matrix.artifact.pr }}
+ override_branch: ${{ matrix.artifact.branch }}
+ token: ${{ secrets.CODECOV_TOKEN }}
+ fail_ci_if_error: true
+
+ # Each PR is recorded from its own upload jobs, so a failed upload, whether
a transient error or
+ # an artifact that never passes extraction, leaves only its own PR
unrecorded. The next sweep
+ # uploads that PR again; re-uploading the same report for a commit is
harmless.
+ record:
+ needs: [select, upload]
+ if: needs.select.result == 'success' && needs.select.outputs.prs != '[]'
&& !cancelled()
+ name: Record the uploaded runs
+ runs-on: ubuntu-slim
+ timeout-minutes: 10
+ permissions:
+ actions: read
+ checks: write
+ steps:
+ - name: Create the checks
+ uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3
# v9.0.0
+ env:
+ ARTIFACTS: ${{ needs.select.outputs.artifacts }}
+ PRS: ${{ needs.select.outputs.prs }}
+ with:
+ script: |
+ const CHECK = 'Coverage (fork)';
+ const upstream = { owner: context.repo.owner, repo:
context.repo.repo };
+ const artifacts = JSON.parse(process.env.ARTIFACTS);
+ const forks = new Map(artifacts.map((item) => [item.pr,
item.fork]));
+ // Upload job names end with the artifact id; take the latest
attempt of each job.
+ const jobs = await
github.paginate(github.rest.actions.listJobsForWorkflowRun, {
+ ...upstream, run_id: context.runId, filter: 'latest', per_page:
100,
+ });
+ const uploaded = new Set(jobs.filter((job) => job.conclusion ===
'success')
+ .map((job) => /^Upload #\d+ .*
\((\d+)\)$/.exec(job.name)?.[1]).filter(Boolean));
+ const errors = [];
+ for (const entry of JSON.parse(process.env.PRS)) {
+ const own = artifacts.filter((item) => item.pr === entry.pr);
+ if (!own.every((item) =>
uploaded.has(String(item.artifact_id)))) {
+ core.warning('#' + entry.pr + ': an upload failed, so the next
sweep uploads this PR again');
+ continue;
+ }
+ const runUrl = (run) => 'https://github.com/' +
forks.get(entry.pr) + '/actions/runs/' + run.run_id;
+ const runIds = entry.runs.map((run) => run.run_id + '.' +
run.run_attempt);
+ try {
+ await github.rest.checks.create({
+ ...upstream,
+ name: CHECK,
+ head_sha: entry.sha,
+ external_id: 'hudi-fork-coverage:' + entry.pr + ':' +
runIds.join(':'),
+ status: 'completed',
+ conclusion: 'success',
+ details_url: runUrl(entry.runs[0]),
+ output: {
+ title: 'Fork coverage uploaded to Codecov',
+ summary: [
+ 'Uploaded the coverage reports of these fork runs on ' +
forks.get(entry.pr) + ' for ' +
+ entry.sha + ' to Codecov:\n' + entry.runs.map((run) =>
+ '- [' + run.workflow + ' run ' + run.run_id + ',
attempt ' + run.run_attempt + '](' +
+ runUrl(run) + ')').join('\n'),
+ 'This coverage is reported by the contributor\'s fork
runs, so it is self-reported. ' +
+ 'The merge queue runs the authoritative CI on ASF
runners.',
+ ].join('\n\n'),
+ },
+ });
+ core.info('#' + entry.pr + ': recorded runs ' + runIds.join('
and '));
+ } catch (error) {
+ // An unrecorded PR is uploaded again by the next sweep.
+ errors.push('#' + entry.pr + ': ' + error.message);
+ }
+ }
+ if (errors.length) core.setFailed(errors.join('\n'));
diff --git a/.github/workflows/java_ci_engines.yml
b/.github/workflows/java_ci_engines.yml
index 6e5d81fd7fa6..b6e629b4f936 100644
--- a/.github/workflows/java_ci_engines.yml
+++ b/.github/workflows/java_ci_engines.yml
@@ -145,10 +145,8 @@ jobs:
run: ./scripts/jacoco/generate_merged_coverage_report.sh
$GITHUB_WORKSPACE
- name: Upload coverage to Codecov
if: always() && endsWith(matrix.flinkProfile, '2.2') &&
github.event_name != 'merge_group'
- uses: codecov/codecov-action@75cd11691c0faa626561e295848008c8a7dddffe
# v5
+ uses: ./.github/actions/upload-coverage
with:
- files: ./jacoco-report.xml
- disable_search: true
flags: flink-integration-tests
token: ${{ secrets.CODECOV_TOKEN }}
@@ -193,10 +191,8 @@ jobs:
run: ./scripts/jacoco/generate_merged_coverage_report.sh
$GITHUB_WORKSPACE
- name: Upload coverage to Codecov
if: always() && endsWith(matrix.flinkProfile, '2.2') &&
github.event_name != 'merge_group'
- uses: codecov/codecov-action@75cd11691c0faa626561e295848008c8a7dddffe
# v5
+ uses: ./.github/actions/upload-coverage
with:
- files: ./jacoco-report.xml
- disable_search: true
flags: flink-integration-tests
token: ${{ secrets.CODECOV_TOKEN }}
@@ -536,10 +532,8 @@ jobs:
run: ./scripts/jacoco/generate_merged_coverage_report.sh
$GITHUB_WORKSPACE
- name: Upload coverage to Codecov
if: always() && github.event_name != 'merge_group'
- uses: codecov/codecov-action@75cd11691c0faa626561e295848008c8a7dddffe
# v5
+ uses: ./.github/actions/upload-coverage
with:
- files: ./jacoco-report.xml
- disable_search: true
flags: integration-tests
token: ${{ secrets.CODECOV_TOKEN }}
diff --git a/.github/workflows/validate_source.yml
b/.github/workflows/validate_source.yml
index 7cd33498bd5b..918b4f18a5ab 100644
--- a/.github/workflows/validate_source.yml
+++ b/.github/workflows/validate_source.yml
@@ -69,18 +69,18 @@ jobs:
exit 1
fi
- # Seconds of node tests of the fork CI status bridge and of the bot.yml
and
+ # Seconds of node tests of the fork CI status and coverage bridges and
of the bot.yml and
# java_ci_engines.yml routing they check, so they run on every PR
instead of path-filtering on
- # the three workflow files. Unlike the PR metadata checks they need no
PR data, so they also
- # run on merge-queue groups and pushes. The tests read the workflow
files through PyYAML. A
- # pip install into the runner's system Python can be refused (PEP 668),
so the step gets its
- # own Python.
- - name: Set up Python for the fork_ci_status.yml tests
+ # the workflow files. Unlike the PR metadata checks they need no PR
data, so they also run on
+ # merge-queue groups and pushes. The tests read the workflow files
through PyYAML. A pip
+ # install into the runner's system Python can be refused (PEP 668), so
the step gets its own
+ # Python.
+ - name: Set up Python for the fork CI bridge tests
if: ${{ !cancelled() }}
uses: actions/setup-python@v6
with:
python-version: '3.12'
- - name: Test fork_ci_status.yml
+ - name: Test fork CI bridge scripts
if: ${{ !cancelled() }}
run: |
for attempt in 1 2 3; do
@@ -88,7 +88,7 @@ jobs:
if [ "$attempt" -eq 3 ]; then exit 1; fi
sleep 5
done
- node --test .github/ci/scripts/test-fork-ci-status.cjs
+ node --test .github/ci/scripts/test-fork-ci-status.cjs
.github/ci/scripts/test-fork-coverage-bridge.cjs
- name: Check for legacy PR title format using JIRA or MINOR
id: check-legacy-format