On Fri Jul 17, 2026 at 6:14 PM CEST, Jelte Fennema-Nio wrote:
Something like the attached, all done Claude. I'm gonna enjoy my weekend now though. Feel free to take this over.
Changed it a bit and tried it out. This works pretty well and seems like a nice improvement for people that actually backpatch things. I think I like this simplicity a lot better than the patch I submitted earlier to run CI on PRs.
From 629ce282b36fb1ac28e273abf452427d3d3d8418 Mon Sep 17 00:00:00 2001 From: Jelte Fennema-Nio <[email protected]> Date: Fri, 17 Jul 2026 18:08:54 +0200 Subject: [PATCH v2] ci: Make stable branch ccache usable by branches based on them GitHub Actions only lets a workflow run restore caches that were created on the run's own branch or on the repository's default branch. A branch based on e.g. REL_19_STABLE therefore starts compiling from scratch (or from master's cache, which mostly misses for stable branch sources), even when REL_19_STABLE itself was recently built by CI. That makes iterating on backpatches unnecessarily slow. Caches created on the default branch, however, are readable from every branch. So this adds a `workflow_dispatch` trigger with a `branch` input, which allows dispatching the workflow on master while checking out and building another branch: gh workflow run pg-ci.yml --ref master -f branch=REL_19_STABLE Such a run publishes the resulting ccache in master's globally readable cache scope, under keys naming the branch that was built (e.g. `ccache:linux-meson-64:REL_19_STABLE:`). On the restore side, every CI run now derives which stable branch its sources belong to from the version in `meson.build`, and prefers that branch's shared cache over master's when restoring the base cache. So after a single dispatch for REL_19_STABLE, any push of a branch based on it starts out with a warm cache instead of a cold one. --- .github/workflows/pg-ci.yml | 90 +++++++++++++++++++++++++++++++++---- src/tools/ci/README | 24 ++++++++++ 2 files changed, 106 insertions(+), 8 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index 5bc5292d2a5..3b682e4db6e 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -15,6 +15,26 @@ on: # adding the 'pull_request' event, as naively doing so would often lead to # running CI twice. + # Build the ccache for another branch and publish it in the dispatching + # branch's cache scope. GitHub Actions only lets a workflow run restore + # caches created on its own branch or on the default branch, so plain + # pushes of branches based on a stable branch normally can't reuse that + # stable branch's compilation results. But caches created on the default + # branch are readable from everywhere. Thus, dispatching this workflow on + # master with e.g. REL_19_STABLE as the input makes that branch's ccache + # available to all branches (see the 'Derive cache base branch' step for + # the restore side). The run simply executes all of CI on the checked out + # branch, so it doubles as a full test of that branch. Re-dispatch + # occasionally to refresh the cache as the branch progresses. + workflow_dispatch: + inputs: + branch: + description: >- + Branch to check out and build, instead of the branch the workflow + was dispatched on (e.g. REL_19_STABLE) + type: string + required: false + # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. permissions: @@ -26,8 +46,15 @@ concurrency: # neither want to wait for prior runs, nor to cancel them, so that each # separately pushed commit is tested. We achieve that by setting a unique # concurrency group when on such a branch. + # + # workflow_dispatch cache builds (see `on:`) are keyed by the branch they + # build, so that a re-dispatch for a branch replaces a still-running + # build of it, while builds for different branches run concurrently. The + # bare branch name cannot collide with the push groups, which use the + # full refs/heads/... ref. group: | ${{github.workflow }}-${{ + inputs.branch || case(github.ref == 'refs/heads/master' || (startsWith(github.ref, 'refs/heads/REL_') && endsWith(github.ref, '_STABLE')), github.run_id, @@ -139,10 +166,15 @@ env: # Note that we need to be careful to use a separator that can't be in branch # names, otherwise e.g. caches for 'master' might be restored on the # 'master-pending' branch. + # + # For workflow_dispatch runs with a `branch` input, the key names that + # branch rather than the one the workflow was dispatched on: the cache + # describes the sources that were built, not the ref the run executed on + # (which determines the cache's scope, see `on:` above). CACHE_PREFIX_DEFAULT: >- :${{ github.job }}:${{ github.event.repository.default_branch }}: CACHE_PREFIX_BRANCH: >- - :${{ github.job }}:${{ github.ref_name }}: + :${{ github.job }}:${{ inputs.branch || github.ref_name }}: CACHE_SUFFIX: >- ${{ github.run_id }}:${{ github.run_attempt }} @@ -292,25 +324,61 @@ jobs: uses: actions/checkout@v6 with: fetch-depth: ${{ env.CLONE_DEPTH }} - - # We restore both the ccache from the default branch (typically master), - # and from the current branch. This will often allow feature branches to - # start out with a high cache hit ratio. + # Empty for everything but workflow_dispatch runs with a `branch` + # input, and empty means: check out the triggering commit. + ref: ${{ inputs.branch || '' }} + + # Determine which stable branch, if any, the checked out sources + # belong to, by parsing the version from meson.build (a 'devel' + # version means master). The base branch restore below prefers this + # over the default branch's cache: a stable branch's ccache can be + # published in the globally readable default branch cache scope by + # dispatching this workflow, see `on:` above. This is what makes + # plain pushes of branches based on a stable branch start out with + # that branch's cache, rather than master's, which mostly misses for + # stable branch sources. + - &derive_cache_base_step + name: Derive cache base branch + id: cache-base + shell: bash + run: | + version=$(sed -n "s/^ *version: '\([^']*\)'.*/\1/p" meson.build | head -n 1) + restore_key='' + case "$version" in + *devel | '') + ;; + *) + major=${version%%[!0-9]*} + restore_key="ccache:${GITHUB_JOB}:REL_${major}_STABLE:" + ;; + esac + echo "version: $version" + echo "restore-key=$restore_key" | tee -a "$GITHUB_OUTPUT" + + # We restore both the ccache from the base branch (the stable branch + # the sources belong to if its shared cache exists, the default + # branch otherwise), and from the current branch. This will often + # allow feature branches to start out with a high cache hit ratio. # # With ccache it turns out to work to just restore two caches into the # same directory, as it's basically a content addressed store. Stats # could be corrupted, but we zero them out anyway. - &ccache_restore_default_step - name: "ccache: Restore for default branch ${{github.event.repository.default_branch}}" + name: "ccache: Restore for base branch" if: ${{ env.ON_DEFAULT_BRANCH == 'false' }} uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} key: ccache${{env.CACHE_PREFIX_DEFAULT}}${{env.CACHE_SUFFIX}} - restore-keys: ccache${{env.CACHE_PREFIX_DEFAULT}} + # restore-keys are tried in order and only the first match is + # restored, so the derived stable branch cache (empty entries are + # ignored) takes precedence over the default branch's. + restore-keys: | + ${{ steps.cache-base.outputs.restore-key }} + ccache${{env.CACHE_PREFIX_DEFAULT}} - &ccache_restore_branch_step - name: "ccache: Restore for branch ${{ github.ref_name }}" + name: "ccache: Restore for branch ${{ inputs.branch || github.ref_name }}" id: ccache-restore-branch uses: actions/cache/restore@v5 with: @@ -493,6 +561,7 @@ jobs: - *nix_sysinfo_step - *checkout_step + - *derive_cache_base_step - *ccache_restore_default_step - *ccache_restore_branch_step - *linux_prepare_workspace_step @@ -555,6 +624,7 @@ jobs: - *nix_sysinfo_step - *checkout_step + - *derive_cache_base_step - *ccache_restore_default_step - *ccache_restore_branch_step - *linux_prepare_workspace_step @@ -645,6 +715,7 @@ jobs: - *nix_sysinfo_step - *checkout_step + - *derive_cache_base_step - *ccache_restore_default_step - *ccache_restore_branch_step - *linux_prepare_workspace_step @@ -795,6 +866,7 @@ jobs: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mp-key.outputs.key }} + - *derive_cache_base_step - *ccache_restore_default_step - *ccache_restore_branch_step @@ -1116,6 +1188,7 @@ jobs: shell: cmd run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} + - *derive_cache_base_step - *ccache_restore_default_step - *ccache_restore_branch_step @@ -1174,6 +1247,7 @@ jobs: steps: - *nix_sysinfo_step - *checkout_step + - *derive_cache_base_step - *ccache_restore_default_step - *ccache_restore_branch_step diff --git a/src/tools/ci/README b/src/tools/ci/README index 642e518c296..dcab23e02be 100644 --- a/src/tools/ci/README +++ b/src/tools/ci/README @@ -43,6 +43,30 @@ https://github.com/<username>/<reponame>/settings/variables/actions and create a new repository variable named PG_CI_ENABLED, with the value 1. +Building shared caches for stable branches +========================================== + +GitHub Actions only allows a workflow run to restore compilation caches +(ccache) created on the run's own branch or on the repository's default +branch. A branch based on e.g. REL_19_STABLE therefore normally starts +building from scratch (or from the default branch's cache, which mostly +misses), even when REL_19_STABLE itself was recently built by CI. This +makes iterating on backpatches unnecessarily slow. + +However, caches created by runs on the default branch are readable from +every branch. To take advantage of that, the CI workflow can be manually +dispatched on the default branch with the name of another branch as input: + + gh workflow run pg-ci.yml --ref master -f branch=REL_19_STABLE + +Such a run checks out and builds the given branch, while storing the +resulting caches in the default branch's cache scope. CI runs detect which +stable branch the sources they're testing belong to, based on the version +in meson.build, and restore that branch's shared cache if it exists. Since +the run executes the whole CI workflow, it also functions as a full test of +the given branch. + + Viewing CI results in a GitHub repository ========================================= -- 2.54.0
