yuqi1129 commented on code in PR #13049: URL: https://github.com/apache/gravitino/pull/13049#discussion_r4345267337
########## .github/workflows/fork-ci-status.yml: ########## @@ -0,0 +1,157 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. +# +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +name: Fork CI Status + +on: + pull_request_target: + types: [opened, synchronize, reopened] + +permissions: + actions: read + checks: write + contents: read + +jobs: + notify: + name: Notify fork CI + if: github.repository == 'apache/gravitino' + runs-on: ubuntu-slim + steps: + - name: Create fork CI check + uses: actions/github-script@v7 + with: + github-token: ${{ secrets.GITHUB_TOKEN }} + retries: 3 + script: | + const pr = context.payload.pull_request; + if (!pr.head.repo) { + core.warning('The pull request head repository no longer exists.'); + return; + } + const owner = pr.head.repo.owner.login; + const repo = pr.head.repo.name; + const branch = pr.head.ref; + const headSha = pr.head.sha; + const workflow = 'fork-ci.yml'; + + // Pull requests to a branch without Fork CI, such as a release branch it has not + // been backported to, still run that branch's own pull_request workflows. + try { + await github.rest.repos.getContent({ + owner: context.repo.owner, repo: context.repo.repo, + path: `.github/workflows/${workflow}`, ref: pr.base.ref }); + } catch (error) { + if (error.status !== 404) throw error; + core.info(`${pr.base.ref} has no Fork CI; no check is created.`); + return; + } + + let run; + for (let attempt = 0; attempt < 3 && !run; attempt++) { + if (attempt > 0) { + await new Promise(resolve => setTimeout(resolve, 3000)); + } + try { + const runs = await github.paginate( + 'GET /repos/{owner}/{repo}/actions/workflows/{workflow_id}/runs', + { owner, repo, workflow_id: workflow, branch, per_page: 100 }); + run = runs.find(candidate => candidate.head_sha === headSha); + } catch (error) { + core.warning(`Unable to read fork Actions: ${error.message}`); + } + } + + // The check mirrors the Required CI job rather than the run conclusion: a run whose + // suites were all skipped (no pull request yet, or Fork CI disabled in the fork) + // still concludes as success. Failed jobs are listed while the run is still going, + // so reviewers and contributors can open their logs without waiting for the run. + const mirror = async (owner, repo, run) => { + const jobs = await github.paginate( + 'GET /repos/{owner}/{repo}/actions/runs/{run_id}/attempts/{attempt_number}/jobs', + { owner, repo, run_id: run.id, attempt_number: run.run_attempt, per_page: 100 }); + const failed = jobs.filter(job => + job.name !== 'Required CI' && ['failure', 'timed_out'].includes(job.conclusion)); + const failedNote = failed.length + ? `Failed jobs:\n${failed.map(job => `- [${job.name}](${job.html_url})`).join('\n')}` + : ''; + if (run.status !== 'completed') { + // A run stays queued while any of its jobs waits for a runner, even after other + // jobs have finished, so report it as in progress once any job has started. + const started = jobs.some(job => ['in_progress', 'completed'].includes(job.status)); + return { + status: started ? 'in_progress' : 'queued', + title: failed.length ? `Running, ${failed.length} job(s) failed so far` : 'Running', + note: failedNote + }; + } + if (run.conclusion !== 'success') { + // Check runs reject some run conclusions, such as startup_failure. + const conclusion = + ['action_required', 'cancelled', 'failure', 'timed_out'].includes(run.conclusion) + ? run.conclusion + : (['neutral', 'skipped'].includes(run.conclusion) ? 'action_required' : 'failure'); + return { + status: 'completed', conclusion, + title: failed.length ? `${failed.length} job(s) failed` : `Fork CI ${run.conclusion}`, + note: [`The fork workflow run concluded with ${run.conclusion}.`, failedNote] + .filter(Boolean).join('\n\n') + }; + } + const required = jobs.find(job => job.name === 'Required CI'); Review Comment: Addressed in the fixes through [effa17fc17](https://github.com/apache/gravitino/commit/effa17fc1779a2b2a96a8980778a74fb0c3b07f3). Before accepting a fork result, the publisher compares the immutable Git tree SHAs for `.github/workflows`, `.github/actions`, `.github/scripts`, and `.asf.yaml` with the exact PR base commit. It also requires successful jobs from all 16 callers and a successful final job encoding the exact head, base ref and base SHA. Changed CI definitions require an explicit committer-dispatched upstream run using reviewed default-branch workflows. Added regressions for fabricated results and changed definitions. The publisher no longer restores Actions caches: neither verification decisions nor refresh scheduling can be supplied by a contributor-writable cache. ########## .github/workflows/asf-allowlist-check.yml: ########## @@ -20,15 +20,13 @@ name: "ASF Allowlist Check" on: - workflow_dispatch: + # Also run upstream for pull requests that change workflows or actions, so the Review Comment: Addressed with a separate default-branch `Trusted ASF Allowlist` workflow, and added `ASF Allowlist Check` to the required contexts in `.asf.yaml`. I used an unprivileged PR signal followed by `workflow_run`: [ASF's current policy](https://infra.apache.org/github-actions-policy.html#avoid-using-the-pull-request-target-trigger) disables `pull_request_target` from November 2026. The trusted scanner checks out only the default branch, downloads the exact PR head's bounded YAML files as data outside that checkout, and runs the Apache allowlist action. It never checks out or executes PR code, and rechecks the current PR head before publishing the check. End-to-end activation still needs the default-branch rollout. ########## .github/workflows/fork-ci.yml: ########## @@ -0,0 +1,310 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# Run the complete CI entry point from branch pushes in the contributor's repository. +name: Fork CI + +on: + push: + branches: ['**'] + +concurrency: + group: fork-ci-${{ github.ref }} + cancel-in-progress: true + +jobs: + eligibility: + name: Resolve fork CI eligibility + # Downstream mirrors can set this repository variable to keep the + # inherited workflow completely dormant. + if: vars.DISABLE_FORK_CI != 'true' + runs-on: ubuntu-slim + outputs: + applicable: ${{ steps.resolve.outputs.applicable }} + valid: ${{ steps.resolve.outputs.valid == 'true' || steps.merge.outputs.valid == 'true' }} + steps: + - name: Resolve repository and pull request + id: resolve + env: + GH_TOKEN: ${{ github.token }} + HEAD_BRANCH: ${{ github.ref_name }} + HEAD_OWNER: ${{ github.repository_owner }} + HEAD_REPOSITORY: ${{ github.repository }} + HEAD_SHA: ${{ github.sha }} + REPOSITORY_FORK: ${{ github.event.repository.fork }} + run: | + set -euo pipefail + + echo "applicable=false" >> "${GITHUB_OUTPUT}" + echo "valid=false" >> "${GITHUB_OUTPUT}" + + if [[ "${HEAD_REPOSITORY}" == "apache/gravitino" && \ + ("${GITHUB_REF}" == "refs/heads/main" || "${GITHUB_REF}" == refs/heads/branch-*) ]]; then + echo "applicable=true" >> "${GITHUB_OUTPUT}" + echo "base_ref=${GITHUB_REF_NAME}" >> "${GITHUB_OUTPUT}" + echo "valid=true" >> "${GITHUB_OUTPUT}" + exit 0 + fi + + if [[ "${HEAD_REPOSITORY}" != "apache/gravitino" && "${REPOSITORY_FORK}" != "true" ]]; then + echo "This repository is not a fork; it runs its own CI and Fork CI is dormant." + exit 0 + fi + + repository_response="$(curl --fail-with-body --silent --show-error --retry 3 --retry-all-errors --retry-delay 5 \ + --header "Accept: application/vnd.github+json" \ + --header "Authorization: Bearer ${GH_TOKEN}" \ + --header "X-GitHub-Api-Version: 2022-11-28" \ + "${GITHUB_API_URL}/repos/${HEAD_REPOSITORY}")" + source_repository="$(jq -r '.source.full_name // empty' <<< "${repository_response}")" + + if [[ "${HEAD_REPOSITORY}" != "apache/gravitino" && \ + "${source_repository}" != "apache/gravitino" ]]; then + echo "This repository is not in the apache/gravitino fork network; Fork CI is dormant." + exit 0 + fi + + echo "applicable=true" >> "${GITHUB_OUTPUT}" + matches="[]" + match_count=0 + # Give the contributor a few minutes to open the pull request after pushing. + for attempt in $(seq 1 12); do + response="$(curl --fail-with-body --silent --show-error --retry 3 --retry-all-errors --retry-delay 5 \ + --header "Accept: application/vnd.github+json" \ + --header "Authorization: Bearer ${GH_TOKEN}" \ + --header "X-GitHub-Api-Version: 2022-11-28" \ + --get \ + --data-urlencode "state=open" \ + --data-urlencode "head=${HEAD_OWNER}:${HEAD_BRANCH}" \ + --data-urlencode "per_page=10" \ + "https://api.github.com/repos/apache/gravitino/pulls";)" + matches="$(jq -r \ + --arg head_sha "${HEAD_SHA}" \ + --arg head_repository "${HEAD_REPOSITORY}" \ + '[.[] | select(.head.sha == $head_sha and .head.repo.full_name == $head_repository)]' \ + <<< "${response}")" + match_count="$(jq -r 'length' <<< "${matches}")" + if [[ "${match_count}" == "1" ]]; then + break + fi + if [[ "${match_count}" != "0" || "${attempt}" == "12" ]]; then + break + fi + sleep 15 + done + + if [[ "${match_count}" == "0" && "${HEAD_REPOSITORY}" == "apache/gravitino" ]]; then + # Branches in the upstream repository only run for pull requests. + echo "No open pull request for ${HEAD_BRANCH}; Fork CI is dormant." + echo "applicable=false" >> "${GITHUB_OUTPUT}" + exit 0 + fi + + api() { + curl --fail-with-body --silent --show-error --retry 3 --retry-all-errors --retry-delay 5 \ + --header "Accept: application/vnd.github+json" \ + --header "Authorization: Bearer ${GH_TOKEN}" \ + --header "X-GitHub-Api-Version: 2022-11-28" \ + "https://api.github.com/repos/apache/gravitino$1"; + } + + if [[ "${match_count}" == "0" ]]; then Review Comment: Fixed. Eligibility now resolves the head SHA, base ref and base SHA once and passes that immutable context through every reusable suite. The final Required CI job includes all three values, and the publisher requires an exact match with the current PR. A pre-PR run against another base cannot pass. The PR signal now includes `edited`, so retargeting wakes the trusted publisher. It withholds success for a mismatched context; the author must push again or obtain a matching trusted run. Added exact-context and publication-race regression tests. ########## .github/workflows/fork-ci-status-update.yml: ########## @@ -0,0 +1,299 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. +# +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +name: Update Fork CI Status + +on: + schedule: + - cron: '*/5 * * * *' + workflow_dispatch: + +permissions: + actions: read + checks: write + contents: read + pull-requests: read + +concurrency: + group: fork-ci-status-update + cancel-in-progress: false + +jobs: + update: + name: Update fork CI checks + if: github.repository == 'apache/gravitino' + runs-on: ubuntu-slim + steps: + # The state of the previous sweep lets this sweep skip pull requests that have a + # final result and no new activity, without spending API calls on them. + - name: Restore previous sweep state + uses: actions/cache/restore@v4 + with: + path: .fork-ci-status + key: fork-ci-status-${{ github.run_id }}-${{ github.run_attempt }} + restore-keys: fork-ci-status- + + - name: Synchronize fork CI checks + uses: actions/github-script@v7 + with: + github-token: ${{ secrets.GITHUB_TOKEN }} + retries: 3 + script: | + const fs = require('fs'); + const stateFile = '.fork-ci-status/state.json'; + const HOUR = 60 * 60 * 1000; + // A pull request with activity in this window is always checked, which covers + // the time between a push and the check created by the Fork CI Status workflow. + const RECENT_ACTIVITY = HOUR; + // How long a completed check is still watched for a re-run of the fork workflow. + // action_required is also set when the fork run had not started when the check was created. + const WATCH = { action_required: 24 * HOUR, default: 7 * 24 * HOUR }; + // Pull requests without a running check are checked at most this often, so sweeping + // every 5 minutes spends API calls mainly on the checks that are about to change. + const IDLE_INTERVAL = 15 * 60 * 1000; + const FINAL = ['success', 'neutral', 'skipped']; + + const rateLimit = async () => (await github.rest.rateLimit.get()).data.resources.core; + const budgetBefore = await rateLimit(); + let calls = 0; + github.hook.before('request', () => { calls++; }); + + let previous = {}; + try { + previous = JSON.parse(fs.readFileSync(stateFile, 'utf8')); + } catch (error) { + core.info('No previous sweep state; checking every pull request.'); + } + const next = {}; + const now = Date.now(); + + // The check mirrors the Required CI job rather than the run conclusion: a run whose + // suites were all skipped (no pull request yet, or Fork CI disabled in the fork) + // still concludes as success. Failed jobs are listed while the run is still going, + // so reviewers and contributors can open their logs without waiting for the run. + const mirror = async (owner, repo, run) => { + const jobs = await github.paginate( + 'GET /repos/{owner}/{repo}/actions/runs/{run_id}/attempts/{attempt_number}/jobs', + { owner, repo, run_id: run.id, attempt_number: run.run_attempt, per_page: 100 }); + const failed = jobs.filter(job => + job.name !== 'Required CI' && ['failure', 'timed_out'].includes(job.conclusion)); + const failedNote = failed.length + ? `Failed jobs:\n${failed.map(job => `- [${job.name}](${job.html_url})`).join('\n')}` + : ''; + if (run.status !== 'completed') { + // A run stays queued while any of its jobs waits for a runner, even after other + // jobs have finished, so report it as in progress once any job has started. + const started = jobs.some(job => ['in_progress', 'completed'].includes(job.status)); + return { + status: started ? 'in_progress' : 'queued', + title: failed.length ? `Running, ${failed.length} job(s) failed so far` : 'Running', + note: failedNote + }; + } + if (run.conclusion !== 'success') { + // Check runs reject some run conclusions, such as startup_failure. + const conclusion = + ['action_required', 'cancelled', 'failure', 'timed_out'].includes(run.conclusion) + ? run.conclusion + : (['neutral', 'skipped'].includes(run.conclusion) ? 'action_required' : 'failure'); + return { + status: 'completed', conclusion, + title: failed.length ? `${failed.length} job(s) failed` : `Fork CI ${run.conclusion}`, + note: [`The fork workflow run concluded with ${run.conclusion}.`, failedNote] + .filter(Boolean).join('\n\n') + }; + } + const required = jobs.find(job => job.name === 'Required CI'); + if (required && required.conclusion === 'success') { + return { status: 'completed', conclusion: 'success', title: 'All required suites passed' }; + } + return { + status: 'completed', conclusion: 'action_required', + title: 'No suites were verified', + note: `Required CI was ${required ? required.conclusion : 'not run'}, so no suites were verified. ` + + 'Open the pull request first and check that Fork CI is enabled in the fork, then push the branch again.' + }; + }; + const NOT_AVAILABLE = 'Enable GitHub Actions in the fork, merge or rebase the branch onto the latest ' + + 'base branch so it contains fork-ci.yml, and push it again.'; + + const skippable = (pr, state) => { Review Comment: Fixed by removing persistent cached state and the `skippable` path entirely. Scheduled refreshes continue visiting all open PRs, including old successful results, with a rotating starting position and a bounded API budget. Manual dispatch with a PR number refreshes that PR directly. CONTRIBUTING now describes scheduling/queue delays without promising immediate updates. The publisher observes the current run attempt, replaces stale success when a rerun is running or fails, and combines each named job's latest execution across attempts to support partial reruns. Added failed-rerun, manual-refresh and partial-rerun tests. ########## .asf.yaml: ########## @@ -46,7 +46,10 @@ github: protected_branches: main: required_status_checks: - strict: true + strict: false Review Comment: Fixed: restored `strict: true`; the up-to-date requirement is retained. The tested base SHA is now frozen before the suites start, encoded in the final job name, recorded in the upstream check metadata, and compared with the current PR base before success is published. The publisher also rereads the PR immediately before writing the check. CONTRIBUTING and the PR description explicitly state that a base advance requires updating the branch and running CI again. Added base-SHA mismatch and publication-race tests. ########## .github/workflows/fork-ci.yml: ########## @@ -0,0 +1,310 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# Run the complete CI entry point from branch pushes in the contributor's repository. +name: Fork CI + +on: + push: + branches: ['**'] + +concurrency: + group: fork-ci-${{ github.ref }} + cancel-in-progress: true + +jobs: + eligibility: + name: Resolve fork CI eligibility + # Downstream mirrors can set this repository variable to keep the + # inherited workflow completely dormant. + if: vars.DISABLE_FORK_CI != 'true' + runs-on: ubuntu-slim + outputs: + applicable: ${{ steps.resolve.outputs.applicable }} + valid: ${{ steps.resolve.outputs.valid == 'true' || steps.merge.outputs.valid == 'true' }} + steps: + - name: Resolve repository and pull request + id: resolve + env: + GH_TOKEN: ${{ github.token }} + HEAD_BRANCH: ${{ github.ref_name }} + HEAD_OWNER: ${{ github.repository_owner }} + HEAD_REPOSITORY: ${{ github.repository }} + HEAD_SHA: ${{ github.sha }} + REPOSITORY_FORK: ${{ github.event.repository.fork }} + run: | + set -euo pipefail + + echo "applicable=false" >> "${GITHUB_OUTPUT}" + echo "valid=false" >> "${GITHUB_OUTPUT}" + + if [[ "${HEAD_REPOSITORY}" == "apache/gravitino" && \ + ("${GITHUB_REF}" == "refs/heads/main" || "${GITHUB_REF}" == refs/heads/branch-*) ]]; then + echo "applicable=true" >> "${GITHUB_OUTPUT}" + echo "base_ref=${GITHUB_REF_NAME}" >> "${GITHUB_OUTPUT}" + echo "valid=true" >> "${GITHUB_OUTPUT}" + exit 0 + fi + + if [[ "${HEAD_REPOSITORY}" != "apache/gravitino" && "${REPOSITORY_FORK}" != "true" ]]; then Review Comment: Fixed. Standalone repositories and unrelated fork networks now resolve `valid=true`, `sync=false` and test their own pushed commit, without fetching or merging an Apache base. Added a resolver regression test for the standalone path. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
