Hi, Andres shared some CI patches in the 'Heads Up: cirrus-ci is shutting down June 1st' thread [1]. Since this thread is already quite long, I wanted to create a new thread for them.
There were 9 patches in the thread and 3 of them already committed: - 0001-ci-Generate-crashlogs-on-Windows.patch: Committed at d374280e183. - 0002-ci-Use-meson-for-most-of-CompilerWarnings-it-s-.patch: Committed at 3ca57291ff. - 0006-ci-Use-optimized-build-for-mingw.patch: Committed at 4075409de0. ---------- These are the uncommitted patches, I rebased them with small changes: - 0003-ci-Make-msys2-install-smaller.patch No functional changes; the commit message is updated (where -> were). - 0004-ci-Make-our-own-msys2-install-from-scratch-inst.patch 'msys2/setup-msys2@66cd2cce69caa17b53920067426061ca1de3a884 # v2.31.1' this SHA points to v2.32.0, it is fixed. Also Jacob's ``` Can we just get rid of the MSYSTEM envvar now? (And the MSYS2_PATH_TYPE, if we replace it with `path-type: inherit` in the action parameters?) See attached -0011. ``` comment [2] is applied. - 0005-squash-or-drop-Use-pacboy-for-shorter-package-n.patch No change. I lean toward dropping this because explicitly naming packages makes it easier to search if something goes wrong. Jacob said it looks nicer with this change [3]. - 0007-ci-get-build-test-concurrency-from-environment.patch I used %NUMBER_OF_PROCESSORS% instead of 'bash -c nproc' to not depend on bash. - 0008-ci-Make-runs-on-overridable.patch Separatd MacPorts caches by runner architecture to avoid restoring incompatible binaries when changing runners. - 0009-ci-Move-to-per-job-naming-for-containers.patch Removed 'options: *linux_container_options' from CompilerWarnings since it is not used. ---------- Rebased and updated patches are attached. Order of the patches has changed because I got them from Andres' working repository. I CCed people from the [1] thread. [1] https://postgr.es/m/iggjozfshwbqpv33x5jqwtju5k5zrkyu3257dlifxkhtpg7eoq%40k2ccyrdi5dtu [2] https://postgr.es/m/CAOYmi%2BkOGycb6Vw-Z19%3DM8js6d36%3D6%3DkUw4Cu5%2BueQej77ig%2Bg%40mail.gmail.com [3] https://postgr.es/m/CAOYmi%2BnEqAYKjh1r7HiKQ%3DLhFU5LMHw%3DXHuEkUr9We6b0judsg%40mail.gmail.com -- Regards, Nazir Bilal Yavuz Microsoft
From 683cbbf882abb00d3cf32735570466a5b4f8ff3f Mon Sep 17 00:00:00 2001 From: Andres Freund <[email protected]> Date: Mon, 22 Jun 2026 12:20:52 -0400 Subject: [PATCH v14 1/6] ci: get build/test concurrency from environment Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Author: Andres Freund <[email protected]> Co-authored-by: Nazir Bilal Yavuz <[email protected]> Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 44 +++++++++++++++++++++++-------------- 1 file changed, 27 insertions(+), 17 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index e8eb49b5f15..19bb0048506 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,18 @@ jobs: - name: Sysinfo run: | + echo Number of logical processors: %NUMBER_OF_PROCESSORS% + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1203,7 +1213,7 @@ jobs: build echo "::endgroup::" - ninja -C build --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1281,7 +1291,7 @@ jobs: build echo "::endgroup::" - ninja -C build --quiet -j${{env.BUILD_JOBS}} -k 0 \ + ninja -C build --quiet -k 0 \ headerscheck cpluspluscheck - *ccache_decide_save_step -- 2.47.3
From ade28951f5af0d6eb602e529d1b31b42247f218c Mon Sep 17 00:00:00 2001 From: Andres Freund <[email protected]> Date: Mon, 22 Jun 2026 12:20:52 -0400 Subject: [PATCH v14 2/6] ci: Make msys2 install smaller We only needed git and diffutils installed via pacman because the already installed tools were hidden by the login script resetting PATH. "Fix" that instead by telling msys to leave the old PATH around. Also don't install libbacktrace it is not needed at the moment. Author: Andres Freund <[email protected]> Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index 19bb0048506..34153a9b5da 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -1066,6 +1066,11 @@ jobs: CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 + # We don't want using an msys bash to "hide" all the other already + # installed tools, that would require us to install tools into msys that + # are already available otherwise. + MSYS2_PATH_TYPE: inherit + defaults: run: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' @@ -1096,11 +1101,10 @@ jobs: # MSYS2. It dynamically expands to the correct prefix for the active # shell environment. pacman -S --noconfirm --needed --asdeps \ - git bison flex diffutils \ + bison flex \ ${MINGW_PACKAGE_PREFIX}-ccache \ ${MINGW_PACKAGE_PREFIX}-gcc \ ${MINGW_PACKAGE_PREFIX}-icu \ - ${MINGW_PACKAGE_PREFIX}-libbacktrace \ ${MINGW_PACKAGE_PREFIX}-libxml2 \ ${MINGW_PACKAGE_PREFIX}-libxslt \ ${MINGW_PACKAGE_PREFIX}-lz4 \ -- 2.47.3
From 66bf41f913e6ea764027bb72ca3e3a8cd1d1284c Mon Sep 17 00:00:00 2001 From: Andres Freund <[email protected]> Date: Mon, 22 Jun 2026 12:20:52 -0400 Subject: [PATCH v14 3/6] ci: Make our own msys2 install from scratch, instead of moving Previously we relocated the existing install, for performance reasons. However that has three disadvantages: 1) moving the install and then installing the packages we need is slower than just creating a new install 2) It hardcodes that D: is the fast drive, but that turns out to depend on the type of runner used 3) We were not using any caching and therefore downloaded the same files over and over. We could have added caching in the previous approach too, but it'd have been extra work. I previously prototyped handrolling the install, instead of using msys2/setup-msys2@v2, and that does turn out to be a bit faster, but it doesn't include caching. By the time we add that, the overall complexity seems like it'd be too big. The other alternative would be to generate a downloadable release in the pg-vm-images repo. That'd likely be even faster. Author: Andres Freund <[email protected]> Co-authored-by: Nazir Bilal Yavuz <[email protected]> Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 74 +++++++++++++++++-------------------- 1 file changed, 33 insertions(+), 41 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index 34153a9b5da..9118fde1bd4 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -1055,7 +1055,6 @@ jobs: MSYS: winjitdebug CHERE_INVOKING: 1 - MSYSTEM: UCRT64 # Keep -Dnls explicitly disabled, as the number of files it creates # causes a noticeable slowdown. @@ -1066,55 +1065,48 @@ jobs: CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 - # We don't want using an msys bash to "hide" all the other already - # installed tools, that would require us to install tools into msys that - # are already available otherwise. - MSYS2_PATH_TYPE: inherit - defaults: run: - shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + shell: msys2 {0} steps: - *windows_disable_defender_step - *window_setup_hosts_step - *checkout_step - # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to - # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses - # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. - # - # This reduces the total runtime of this task by ~15 minutes. + # The pre-existing msys install is on C:\, often a rather slow system + # disk, whereas the workspace is often on a faster, ephemeral disk. + # Using the pre-existing msys would often increase the total runtime of + # this task by ~15 minutes. # - # robocopy returns 0-7 on success (with various "files copied" bits - # set) and 8+ on real failure, so we have to translate its exit code. - - name: Relocate MSYS2 to D - shell: pwsh - run: | - robocopy C:\msys64 D:\msys64 /E /NJS /NJH /NFL /NDL /NP - if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } - exit 0 - - - name: Setup MSYS2 - run: | - # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the - # MSYS2. It dynamically expands to the correct prefix for the active - # shell environment. - pacman -S --noconfirm --needed --asdeps \ - bison flex \ - ${MINGW_PACKAGE_PREFIX}-ccache \ - ${MINGW_PACKAGE_PREFIX}-gcc \ - ${MINGW_PACKAGE_PREFIX}-icu \ - ${MINGW_PACKAGE_PREFIX}-libxml2 \ - ${MINGW_PACKAGE_PREFIX}-libxslt \ - ${MINGW_PACKAGE_PREFIX}-lz4 \ - ${MINGW_PACKAGE_PREFIX}-make \ - ${MINGW_PACKAGE_PREFIX}-meson \ - ${MINGW_PACKAGE_PREFIX}-perl \ - ${MINGW_PACKAGE_PREFIX}-pkgconf \ - ${MINGW_PACKAGE_PREFIX}-readline \ - ${MINGW_PACKAGE_PREFIX}-zlib \ - ${MINGW_PACKAGE_PREFIX}-zstd + # Instead we use msys2/setup-msys2 to set up our own install. That's + # faster than moving the install from C:\, and also works when the + # workspace is not on a separate disk. + - name: Install MSYS2 + uses: msys2/setup-msys2@66cd2cce69caa17b53920067426061ca1de3a884 # v2.32.0 + with: + msystem: UCRT64 + # We don't want using an msys bash to "hide" all the other already + # installed tools, that would require us to install tools into msys + # that are already available otherwise. + path-type: inherit + update: true + location: ${{github.workspace}} + install: >- + bison flex + mingw-w64-ucrt-x86_64-ccache + mingw-w64-ucrt-x86_64-gcc + mingw-w64-ucrt-x86_64-icu + mingw-w64-ucrt-x86_64-libxml2 + mingw-w64-ucrt-x86_64-libxslt + mingw-w64-ucrt-x86_64-lz4 + mingw-w64-ucrt-x86_64-make + mingw-w64-ucrt-x86_64-meson + mingw-w64-ucrt-x86_64-perl + mingw-w64-ucrt-x86_64-pkgconf + mingw-w64-ucrt-x86_64-readline + mingw-w64-ucrt-x86_64-zlib + mingw-w64-ucrt-x86_64-zstd - *nix_sysinfo_step -- 2.47.3
From f8b4395a81feb8eaada937f2386272d567ce963a Mon Sep 17 00:00:00 2001 From: Andres Freund <[email protected]> Date: Mon, 22 Jun 2026 12:20:52 -0400 Subject: [PATCH v14 4/6] squash-or-drop: Use pacboy for shorter package names --- .github/workflows/pg-ci.yml | 28 +++++++++++++++------------- 1 file changed, 15 insertions(+), 13 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index 9118fde1bd4..304175d76a5 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -1094,19 +1094,21 @@ jobs: location: ${{github.workspace}} install: >- bison flex - mingw-w64-ucrt-x86_64-ccache - mingw-w64-ucrt-x86_64-gcc - mingw-w64-ucrt-x86_64-icu - mingw-w64-ucrt-x86_64-libxml2 - mingw-w64-ucrt-x86_64-libxslt - mingw-w64-ucrt-x86_64-lz4 - mingw-w64-ucrt-x86_64-make - mingw-w64-ucrt-x86_64-meson - mingw-w64-ucrt-x86_64-perl - mingw-w64-ucrt-x86_64-pkgconf - mingw-w64-ucrt-x86_64-readline - mingw-w64-ucrt-x86_64-zlib - mingw-w64-ucrt-x86_64-zstd + # name:p means MINGW_PACKAGE_PREFIX-only + pacboy: >- + ccache:p + gcc:p + icu:p + libxml2:p + libxslt:p + lz4:p + make:p + meson:p + perl:p + pkgconf:p + readline:p + zlib:p + zstd:p - *nix_sysinfo_step -- 2.47.3
From 9eb244fe9c40a1f14797f9582e7b466419ef4d2c Mon Sep 17 00:00:00 2001 From: Andres Freund <[email protected]> Date: Mon, 22 Jun 2026 12:20:52 -0400 Subject: [PATCH v14 5/6] ci: Make runs-on overridable Previously what runners CI ran on was hardcoded. Unfortunately that makes it harder to use faster or self hosted runners (e.g. in a private repository where the runners are very slow, or self hosted runners for cfbot). Fix that by allowing to override the runner types via repository/organization variables. How to do that is documented in src/tools/ci/README. Testing faster github hosted runners showed that we have some dependencies on specific locations, fix most of those. The MSYS2 installation changes are handled in a separate commit. Author: Andres Freund <[email protected]> Co-authored-by: Nazir Bilal Yavuz <[email protected]> Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 66 +++++++++++++++++++++++-------------- src/tools/ci/README | 21 ++++++++++++ 2 files changed, 63 insertions(+), 24 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index 304175d76a5..e8234fa5848 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -106,12 +106,6 @@ env: --with-uuid=ossp --with-zstd - # Centrally define the version of linux runners, to make it easier to - # update. We don't just want to use ubuntu-latest, as it's not implausible - # there will be breakage when that switches to the next ubuntu version. - _LINUX_RUNS_ON: &linux_runs_on | - ubuntu-24.04 - # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/'. CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/main @@ -154,7 +148,8 @@ jobs: warn-if-not-opted-in: name: Report if not opted into CI if: ${{vars.PG_CI_ENABLED != '1'}} - runs-on: ubuntu-slim + # See the setup task for an explanation + runs-on: ${{ case(vars.PG_CI_RUNS_ON_SETUP != '', vars.PG_CI_RUNS_ON_SETUP, 'ubuntu-slim') }} steps: - name: Warn env: @@ -184,21 +179,43 @@ jobs: # Only run CI if repo owner opted in. If this task is skipped due to the # if, none of it's depending tasks (i.e. the actual CI tasks) run either. if: ${{vars.PG_CI_ENABLED == '1'}} - runs-on: *linux_runs_on timeout-minutes: 1 outputs: + # Are certain tasks enabled? linux: ${{ steps.os.outputs.linux }} macos: ${{ steps.os.outputs.macos }} windows: ${{ steps.os.outputs.windows }} mingw: ${{ steps.os.outputs.mingw }} compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} sanitycheck: ${{ steps.os.outputs.sanitycheck }} + + # What runners to run on: + # + # To allow over-riding what runners jobs use, we allow setting + # repository / org level variables influencing that choice on a + # per-host-operating-system basis. + # + # This also makes it easier to change the version of linux / windows + # used centrally. We don't just want to use ubuntu-latest, as it's not + # implausible there will be breakage when that switches to the next + # ubuntu version. + runs_on_linux: ${{ case(vars.PG_CI_RUNS_ON_LINUX != '', vars.PG_CI_RUNS_ON_LINUX, 'ubuntu-24.04') }} + runs_on_windows: ${{ case(vars.PG_CI_RUNS_ON_WINDOWS != '', vars.PG_CI_RUNS_ON_WINDOWS, 'windows-2022') }} + runs_on_macos: ${{ case(vars.PG_CI_RUNS_ON_MACOS != '', vars.PG_CI_RUNS_ON_MACOS, 'macos-15') }} + # runs_on_setup: an output can't be used as the input of the setup job + # itself, so it's done inline below. + + # Exports: + # # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} + # Can't use the output from above, so repeated verbatim here + runs-on: ${{ case(vars.PG_CI_RUNS_ON_SETUP != '', vars.PG_CI_RUNS_ON_SETUP, 'ubuntu-slim') }} + steps: # Anchor reused by other jobs further down. GitHub Actions supports YAML # anchors/aliases but not merge keys, so the alias copies the whole step @@ -258,7 +275,7 @@ jobs: if: | !cancelled() && needs.setup.outputs.sanitycheck == 'true' - runs-on: *linux_runs_on + runs-on: ${{ needs.setup.outputs.runs_on_linux }} timeout-minutes: 15 container: &linux_ci_container image: ${{ needs.setup.outputs.container_linux_ci }} @@ -446,7 +463,7 @@ jobs: !cancelled() && needs.setup.outputs.linux == 'true' && needs.sanity-check.result != 'failure' - runs-on: *linux_runs_on + runs-on: ${{ needs.setup.outputs.runs_on_linux }} container: *linux_ci_container timeout-minutes: 60 @@ -540,7 +557,7 @@ jobs: name: Linux - Meson (32-bit) needs: [setup, sanity-check] if: *linux_job_if - runs-on: *linux_runs_on + runs-on: ${{ needs.setup.outputs.runs_on_linux }} container: *linux_ci_container timeout-minutes: 60 env: *linux_env @@ -632,7 +649,7 @@ jobs: name: Linux - Meson (64-bit) needs: [setup, sanity-check] if: *linux_job_if - runs-on: *linux_runs_on + runs-on: ${{ needs.setup.outputs.runs_on_linux }} container: *linux_ci_container timeout-minutes: 60 env: *linux_env @@ -688,7 +705,7 @@ jobs: !cancelled() && needs.setup.outputs.macos == 'true' && needs.sanity-check.result != 'failure' - runs-on: macos-15 + runs-on: ${{ needs.setup.outputs.runs_on_macos }} timeout-minutes: 60 env: MACPORTS_CACHE: ${{ github.workspace }}/macports-cache @@ -746,10 +763,10 @@ jobs: macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${{runner.arch}}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" # It's faster to start with a partial cache for the same macos - # version than from scratch - echo "restore-key=macports-${macos_major}-" >> "$GITHUB_OUTPUT" + # version and architecture than from scratch + echo "restore-key=macports-${macos_major}-${{runner.arch}}-" >> "$GITHUB_OUTPUT" - name: "MacPorts: Restore cache" id: mp-restore @@ -845,7 +862,7 @@ jobs: !cancelled() && needs.setup.outputs.windows == 'true' && needs.sanity-check.result != 'failure' - runs-on: windows-2022 + runs-on: ${{ needs.setup.outputs.runs_on_windows }} timeout-minutes: 60 # As described at the top of the task, split the tests across two runners @@ -861,7 +878,7 @@ jobs: env: # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 - PG_REGRESS_SOCK_DIR: 'd:\pgsock' + PG_REGRESS_SOCK_DIR: ${{ github.workspace }}/pgsock TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- @@ -966,7 +983,8 @@ jobs: - name: Install dependencies shell: pwsh run: | - # meson is not preinstalled on windows-2022. Install via pip + # meson is not preinstalled, at least on windows-2022. Install via + # pip echo ::group::pip python -m pip install --upgrade meson if (!$?) { throw 'cmdfail' } @@ -998,7 +1016,7 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} + run: mkdir "${{env.PG_REGRESS_SOCK_DIR}}" - &windows_setup_debugger_step name: Setup Windows debugger @@ -1045,12 +1063,12 @@ jobs: !cancelled() && needs.setup.outputs.mingw == 'true' && needs.sanity-check.result != 'failure' - runs-on: windows-2022 + runs-on: ${{ needs.setup.outputs.runs_on_windows }} timeout-minutes: 60 env: # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 - PG_REGRESS_SOCK_DIR: 'd:\pgsock' + PG_REGRESS_SOCK_DIR: ${{ github.workspace }}/pgsock TAR: "c:/windows/system32/tar.exe" MSYS: winjitdebug @@ -1124,7 +1142,7 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} + run: mkdir "${{env.PG_REGRESS_SOCK_DIR}}" - *windows_setup_debugger_step @@ -1166,7 +1184,7 @@ jobs: !cancelled() && needs.setup.outputs.compilerwarnings == 'true' && needs.sanity-check.result != 'failure' - runs-on: *linux_runs_on + runs-on: ${{ needs.setup.outputs.runs_on_linux }} timeout-minutes: 60 container: image: ${{ needs.setup.outputs.container_linux_ci_docs }} diff --git a/src/tools/ci/README b/src/tools/ci/README index 642e518c296..9276d4a654c 100644 --- a/src/tools/ci/README +++ b/src/tools/ci/README @@ -75,3 +75,24 @@ messages. Currently the following controls are available: Only runs CI on operating systems specified. This can be useful when addressing portability issues affecting only a subset of platforms. + + +Controlling which runners CI uses +================================= + +By default each job runs on a GitHub-hosted runner appropriate for its +operating system. This can be overridden on a per-operating-system basis by +creating repository or organization variables, at +https://github.com/<username>/<reponame>/settings/variables/actions + +This is useful to run on faster GitHub-hosted runners, or on self-hosted +runners. + +The following variables are recognized. If a variable is unset or empty, a +default runner suitable for that operating system is used: + +- PG_CI_RUNS_ON_LINUX: Linux jobs, including SanityCheck and CompilerWarnings +- PG_CI_RUNS_ON_WINDOWS: the Windows (VS) and MinGW jobs +- PG_CI_RUNS_ON_MACOS: the macOS job +- PG_CI_RUNS_ON_SETUP: the lightweight bookkeeping jobs (opt-in warning and + job setup) -- 2.47.3
From 78f7156c18e30b9c7a277f2847bbe5e11a4464ac Mon Sep 17 00:00:00 2001 From: Andres Freund <[email protected]> Date: Mon, 22 Jun 2026 12:20:52 -0400 Subject: [PATCH v14 6/6] ci: Move to per-job naming for containers Previously we only had two types of containers, "ci" for everything but the CompilerWarnings job and ci_docs that had the tools necessary to build the docs as part of CompilerWarnings. But that was unnecessarily heavy for several of the jobs. As the pull time is a noticeable performance factor, that's not great. Instead use containers that are named by their job. Today some of those are the same, but after this change we can optimize the containers for their jobs without a problem. It probably would make sense to allow over-riding CONTAINER_REPO on a per-repo/org basis, but that seems better done as a separate change. Author: Andres Freund <[email protected]> Co-authored-by: Nazir Bilal Yavuz <[email protected]> Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 31 ++++++++++++++++++++----------- 1 file changed, 20 insertions(+), 11 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index e8234fa5848..46fcbc7d1c1 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -108,9 +108,12 @@ env: # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/'. + # + # The containers are named + # ghcr.io/anarazel/pg-vm-images/<branch>/linux_debian_ci/<debian version>/<ci-job>:latest CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/main - CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest - CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest + CONTAINER_LINUX_CI_NAME: linux_debian_ci/trixie + CONTAINER_TAG: latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. @@ -193,7 +196,7 @@ jobs: # # To allow over-riding what runners jobs use, we allow setting # repository / org level variables influencing that choice on a - # per-host-operating-system basis. + # per-host-operating-system basis. See also src/tools/ci/README # # This also makes it easier to change the version of linux / windows # used centrally. We don't just want to use ubuntu-latest, as it's not @@ -210,8 +213,8 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} - container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} + container_linux_ci_base: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_NAME }} + container_tag: ${{ env.CONTAINER_TAG }} # Can't use the output from above, so repeated verbatim here runs-on: ${{ case(vars.PG_CI_RUNS_ON_SETUP != '', vars.PG_CI_RUNS_ON_SETUP, 'ubuntu-slim') }} @@ -277,8 +280,8 @@ jobs: needs.setup.outputs.sanitycheck == 'true' runs-on: ${{ needs.setup.outputs.runs_on_linux }} timeout-minutes: 15 - container: &linux_ci_container - image: ${{ needs.setup.outputs.container_linux_ci }} + container: + image: ${{ needs.setup.outputs.container_linux_ci_base }}/sanity-check:${{ needs.setup.outputs.container_tag }} # Options passed to all linux containers. Not all of the jobs need # all of them, but it's easier to just define them centrally. @@ -464,7 +467,9 @@ jobs: needs.setup.outputs.linux == 'true' && needs.sanity-check.result != 'failure' runs-on: ${{ needs.setup.outputs.runs_on_linux }} - container: *linux_ci_container + container: + image: ${{ needs.setup.outputs.container_linux_ci_base }}/linux-autoconf:${{ needs.setup.outputs.container_tag }} + options: *linux_container_options timeout-minutes: 60 env: &linux_env @@ -558,7 +563,9 @@ jobs: needs: [setup, sanity-check] if: *linux_job_if runs-on: ${{ needs.setup.outputs.runs_on_linux }} - container: *linux_ci_container + container: + image: ${{ needs.setup.outputs.container_linux_ci_base }}/linux-meson-32:${{ needs.setup.outputs.container_tag }} + options: *linux_container_options timeout-minutes: 60 env: *linux_env @@ -650,7 +657,9 @@ jobs: needs: [setup, sanity-check] if: *linux_job_if runs-on: ${{ needs.setup.outputs.runs_on_linux }} - container: *linux_ci_container + container: + image: ${{ needs.setup.outputs.container_linux_ci_base }}/linux-meson-64:${{ needs.setup.outputs.container_tag }} + options: *linux_container_options timeout-minutes: 60 env: *linux_env @@ -1187,7 +1196,7 @@ jobs: runs-on: ${{ needs.setup.outputs.runs_on_linux }} timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.container_linux_ci_docs }} + image: ${{ needs.setup.outputs.container_linux_ci_base }}/compiler-warnings:${{ needs.setup.outputs.container_tag }} env: # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. -- 2.47.3
