Hi, Here's a series implementing a lot, but not all, that I brought up. See the commit messages for details. One change that I hadn't mentioned on here is that I increased the compiler optimizations for mingw, which improves the test runtimes by a enough to even offset the cold-ccache compile times.
I for now used the setup-msys2 action, with a pin to the SHA to address Jacob's concern (even though I still don't see what it buys us). I did try creating an archive of an install, but that ends up with a good bit of additional complexity, because the installations aren't relocatable (there are header paths that are patched during install, apparently). And where the install should be depends on the runner type, larger runners don't have D:, but slower runners require using D:. I think Bilal hacked further on my experiments around this, but I don't quite know where that stands. It's certainly not a trivial change. I'm pretty agnostic whether we want 0005, I just included that because it looks a bit nicer. On 2026-06-12 16:44:25 +0300, Nazir Bilal Yavuz wrote: > On Wed, 10 Jun 2026 at 17:55, Andres Freund <[email protected]> wrote: > > > > Before we can backpatch the CI support I think we need to resolve a few more > > things: > > > > - re-enabling crash reporting for windows, Bilal sent a patch [1] > > > > I think that's pretty much a must have, otherwise debuggin windows issues > > is > > really hard. > > That seems working based on my testing, just needs a review. I'll merge it, I think it's good to go. > > - Cold or inapplicable (e.g. due to a core header change) compiler warnings > > task is very slow (35min). I have a patch that I need to send out to > > convert everything but the headercheck in compilerwarnings to meson, that > > reduces the worst case build times considerably (primarily due to the > > cross > > build being able to use precompiled headers) > > > > I'll try to send that out later today. > > Nice. I encountered this problem a couple of times and they were frustrating. Yea, it's definitely not great as-is. It's a bit weird that with the patch above we still use both meson and autoconf (for headerscheck), but I think it's ok for now. > > - I comparison to cirrus-ci it's considerably more painful (and it wasn't > > exactly pain-free on cirrus either) to access the logs of failed tasks. > > One > > can't just link to the failure or such. > > > > I have wondered about determining which test failed first, and uploading > > the > > most crucial logs for that test separately, so one could at least look and > > link to those without unpacking a .zip. > > I have questions about this. If we do this for all jobs then we can > end up having just too many uploaded files to look at. In this case, > unpacking .zip would be easier to access the logs of failed tasks. I was thinking we'd do this only for the first failure... > > - A decent chunk of test time is spent setting up the containers (I've > > optimized them a bit to reduce that already). Somehow docker is pretty > > slow > > around container extraction. I had already split the containers into one > > for docs and one for the rest, if we did that further, we could make > > startup > > of e.g. sanitycheck (which has an outsized impact) a decent bit faster, > > but > > we can't use the same container for e.g. linux-meson-32. > > > > I think it may be smart to just add per-task tags for the containers. Then > > we can have them initially be the same (by just pushing the same container > > with different tags), which would allow us to adjust the containers > > contents > > later, without needing to patch the workflow in the postgres repo. > > > > I suspect we should do the tag aliases before backpatching, but I'm very > > willing to be convinced otherwise. > > It would make sense to do this before backpatching as some of the > improvements can be done on the images themselves, which can speed up > the process. I did that on the container generation side and now attached a patch to use them. It does reduce the time a decent bit. Greetings, Andres Freund
>From cad64d49bd6bde4db06aa99b9c032844581b7f04 Mon Sep 17 00:00:00 2001 From: Andres Freund <[email protected]> Date: Thu, 11 Jun 2026 09:15:11 -0400 Subject: [PATCH v13a 1/9] ci: Generate crashlogs on Windows cdb.exe is configured to log all crashes to "<workspace>\crashlogs\crashlog-<pid-in-hex>.txt". Upload logs step already configured to collect these logs, so no change is needed on there. Logic is copied from where Postgres CI Windows images are generated [1]. Since this would be too long to include inline in pg-ci.yml, it is implemented as 'src/tools/ci/gha_setup_windows_debugger.ps1' script. [1] https://github.com/anarazel/pg-vm-images/blob/main/scripts/windows_install_dbg.ps1 Author: Nazir Bilal Yavuz <[email protected]> Discussion: https://postgr.es/m/CAN55FZ1BgsXSTzOpehnMa4NzWL8Aivsxx-di7-VT6bZ3j2Omow%40mail.gmail.com --- .github/workflows/pg-ci.yml | 13 ++-- src/tools/ci/gha_setup_windows_debugger.ps1 | 75 +++++++++++++++++++++ 2 files changed, 82 insertions(+), 6 deletions(-) create mode 100644 src/tools/ci/gha_setup_windows_debugger.ps1 diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index 5bc5292d2a5..742a8431782 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -990,6 +990,11 @@ jobs: shell: cmd run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} + - &windows_setup_debugger_step + name: Setup Windows debugger + shell: pwsh + run: src/tools/ci/gha_setup_windows_debugger.ps1 + - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 @@ -1019,9 +1024,6 @@ jobs: call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 run: *meson_test_world_cmd - # TODO: We need to collect crashlogs but for them to be generated, we'd - # have to configure the JIT Debugger to do so. cdb.exe is installed on - # the runner so that is possible. - *upload_logs_step @@ -1116,6 +1118,8 @@ jobs: shell: cmd run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} + - *windows_setup_debugger_step + - *ccache_restore_default_step - *ccache_restore_branch_step @@ -1138,9 +1142,6 @@ jobs: - name: Test world run: *meson_test_world_cmd - # TODO: We want to include crashlogs, but they are not yet - # collected. cdb.exe is installed on the runner, so we can configure it - # appropriately. - *upload_logs_step diff --git a/src/tools/ci/gha_setup_windows_debugger.ps1 b/src/tools/ci/gha_setup_windows_debugger.ps1 new file mode 100644 index 00000000000..babfbe76278 --- /dev/null +++ b/src/tools/ci/gha_setup_windows_debugger.ps1 @@ -0,0 +1,75 @@ +# Setup Windows debugger to log all crashes to +# <workspace>\crashlogs\crashlog-<pid-in-hex>.txt + +$ErrorActionPreference = 'Stop' + +$crashdir = "$env:GITHUB_WORKSPACE/crashlogs" +New-Item -ItemType Directory -Force -Path $crashdir + +# Ensure restricted child processes can write the log file +icacls $crashdir /grant "${env:USERNAME}:(OI)(CI)F" /Q + +# Prevent windows error handling dialog from causing hangs +New-ItemProperty -Force -Path 'HKLM:\SOFTWARE\Microsoft\Windows\Windows Error Reporting' ` + -Name 'DontShowUI' -Value 1 -PropertyType DWord +New-ItemProperty -Force -Path 'HKLM:\SOFTWARE\Microsoft\Windows\Windows Error Reporting' ` + -Name 'Disabled' -Value 1 -PropertyType DWord + +### Fallback minidumps if the JIT debugger below doesn't run +New-Item -Force -Path 'HKLM:\SOFTWARE\Microsoft\Windows\Windows Error Reporting' ` + -Name 'LocalDumps' +New-ItemProperty -Force -Path 'HKLM:\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps' ` + -Name 'DumpFolder' -Value $crashdir -PropertyType ExpandString +New-ItemProperty -Force -Path 'HKLM:\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps' ` + -Name 'DumpCount' -Value 5 -PropertyType DWord +New-ItemProperty -Force -Path 'HKLM:\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps' ` + -Name 'DumpType' -Value 1 -PropertyType DWord +### + +$cdb64 = @( + 'C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\cdb.exe', + 'C:\Program Files\Windows Kits\10\Debuggers\x64\cdb.exe' + ) | Where-Object { Test-Path $_ } | Select-Object -First 1 +$cdb86 = $cdb64.Replace('\x64\', '\x86\') + +### +# -p PID: +# Specifies the decimal process ID to be debugged. This is used to debug a +# process that is already running. +# -e Event: +# Signals the debugger that the specified event has occurred. This option is +# only used when starting the debugger programmatically. +# -g: +# Ignores the initial breakpoint in target application. This option will +# cause the target application to continue running after it is started or +# CDB attaches to it, unless another breakpoint has been set. +# -kqm: +# Starts CDB/NTSD in quiet mode. +# -c "command": +# Specifies the initial debugger command to run at start-up. This command +# must be surrounded with quotation marks. Multiple commands can be +# separated with semicolons. +### +$debuggerArgs = ' -p %ld -e %ld -g -kqm -c ".lines -e; .symfix+ ; aS /x proc $tpid ; .block {.logappend ' + "$crashdir/crashlog-" + '${proc}.txt} ; lsa $ip ; ~*kP ; !peb ; .logclose ; q "' + +Write-Host "Using cdb (x64): $cdb64" +Set-ItemProperty ` + -Path 'HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug' ` + -Name 'Debugger' -Value ('"' + $cdb64 + '"' + $debuggerArgs) +New-ItemProperty -Force -PropertyType DWord -Value 1 ` + -Path 'HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug' ` + -Name 'Auto' + +Write-Host "Using cdb (x86): $cdb86" +Set-ItemProperty ` + -Path 'HKLM:\SOFTWARE\WOW6432Node\Microsoft\Windows NT\CurrentVersion\AeDebug' ` + -Name 'Debugger' -Value ('"' + $cdb86 + '"' + $debuggerArgs) +New-ItemProperty -Force -PropertyType DWord -Value 1 ` + -Path 'HKLM:\SOFTWARE\WOW6432Node\Microsoft\Windows NT\CurrentVersion\AeDebug' ` + -Name 'Auto' + +# Show registered AeDebug values for diagnostics +Get-ItemProperty 'HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug' | + Format-List Debugger,Auto +Get-ItemProperty 'HKLM:\SOFTWARE\WOW6432Node\Microsoft\Windows NT\CurrentVersion\AeDebug' | + Format-List Debugger,Auto -- 2.54.0.450.g9ac3f193c0
>From a31ad0492f02fb55693f2306ab5693985de560d7 Mon Sep 17 00:00:00 2001 From: Andres Freund <[email protected]> Date: Tue, 9 Jun 2026 10:18:50 -0400 Subject: [PATCH v13a 2/9] ci: Use meson for most of CompilerWarnings, it's a lot faster Previously, with a cold cache or a change in a central header, the CompilerWarnings job would take ~35 minutes. The worst aspect of that was the windows cross build, which would take about 8-10 minutes. Migrate the compiler warnings tasks to meson, that generally makes the cold-cache build a bit faster, and makes the windows cross build a *lot* faster, due to being able to use precompiled headers. With that the cold cache performance improves to about 12 minutes. Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 38 +++++++++++++-------------- src/tools/ci/meson-cross-w64-ucrt.txt | 21 +++++++++++++++ 2 files changed, 40 insertions(+), 19 deletions(-) create mode 100644 src/tools/ci/meson-cross-w64-ucrt.txt diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index 742a8431782..81801800dac 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -1171,6 +1171,8 @@ jobs: # compilers / flag combinations. CCACHE_MAXSIZE: "1G" DEFAULT_BUILD: world-bin + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 steps: - *nix_sysinfo_step @@ -1186,53 +1188,56 @@ jobs: - name: gcc warnings + (dtrace) if: ${{ !cancelled() }} env: - CONF: ${{env.LINUX_CONFIGURE_FEATURES}} --cache gcc.cache --enable-dtrace + CONF: ${{env.MESON_COMMON_FEATURES}} -Ddtrace=enabled CC: ccache gcc CXX: ccache g++ CLANG: ccache clang run: &compiler_warnings_cmd | echo "::group::configure" - ./configure \ - ${{env.CONF}} \ - CLANG="ccache clang" + rm -rf build-meson + meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} ${{env.DEFAULT_BUILD}} + ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) if: ${{ !cancelled() }} env: - CONF: ${{env.LINUX_CONFIGURE_FEATURES}} --cache gcc.cache --enable-cassert + CONF: ${{env.MESON_COMMON_FEATURES}} -Dcassert=true CC: ccache gcc CXX: ccache g++ + CLANG: ccache clang run: *compiler_warnings_cmd # clang, cassert off, dtrace off - name: clang warnings if: ${{ !cancelled() }} env: - CONF: ${{env.LINUX_CONFIGURE_FEATURES}} --cache clang.cache + CONF: ${{env.MESON_COMMON_FEATURES}} CC: ccache clang CXX: ccache clang++ + CLANG: ccache clang run: *compiler_warnings_cmd # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) if: ${{ !cancelled() }} env: - CONF: ${{env.LINUX_CONFIGURE_FEATURES}} --cache clang.cache --enable-cassert --enable-dtrace + CONF: ${{env.MESON_COMMON_FEATURES}} -Dcassert=true -Ddtrace=enabled CC: ccache clang CXX: ccache clang++ + CLANG: ccache clang run: *compiler_warnings_cmd - name: mingw warnings (cross compilation) if: ${{ !cancelled() }} env: - CONF: --host=x86_64-w64-mingw32ucrt --enable-cassert --without-icu - CC: ccache x86_64-w64-mingw32ucrt-gcc - CXX: ccache x86_64-w64-mingw32ucrt-g++ + CONF: >- + --cross-file src/tools/ci/meson-cross-w64-ucrt.txt + --buildtype debug + -Db_pch=true + -Dcassert=true run: *compiler_warnings_cmd ### @@ -1241,12 +1246,8 @@ jobs: # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation if: ${{ !cancelled() }} - env: - CONF: --cache gcc.cache - CC: ccache gcc - CXX: ccache g++ - DEFAULT_BUILD: -C doc - run: *compiler_warnings_cmd + run: + ninja -C build-meson docs ### # Verify headerscheck / cpluspluscheck succeed @@ -1261,7 +1262,6 @@ jobs: echo "::group::configure" ./configure \ ${{env.LINUX_CONFIGURE_FEATURES}} \ - --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" diff --git a/src/tools/ci/meson-cross-w64-ucrt.txt b/src/tools/ci/meson-cross-w64-ucrt.txt new file mode 100644 index 00000000000..b6144979416 --- /dev/null +++ b/src/tools/ci/meson-cross-w64-ucrt.txt @@ -0,0 +1,21 @@ +[constants] +triple = 'x86_64-w64-mingw32ucrt' +prefix = '/usr/bin' / triple + '-' + +[binaries] +c = ['ccache', prefix + 'gcc'] +cpp = ['ccache', prefix + 'g++'] +ar = prefix + 'ar' +strip = prefix + 'strip' +pkg-config = prefix + 'pkg-config' +windres = prefix + 'windres' + +[properties] +# Directory that contains 'bin', 'lib', etc +root = '/usr/x86_64-w64-mingw32ucrt' + +[host_machine] +system = 'windows' +cpu_family = 'x86_64' +cpu = 'x86_64' +endian = 'little' -- 2.54.0.450.g9ac3f193c0
>From 81f52c2cc8fe15cb96522da08a43dfa9021f53ae Mon Sep 17 00:00:00 2001 From: Andres Freund <[email protected]> Date: Wed, 10 Jun 2026 23:29:47 -0400 Subject: [PATCH v13a 3/9] ci: Make msys2 install smaller We only needed git and diffutils installed via pacman because the already installed tools where hidden by the login script resetting PATH. "Fix" that instead by telling msys to leave the old PATH around. Also don't install libbacktrace it is not needed at the moment. Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index 81801800dac..442052b72e1 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -1056,6 +1056,11 @@ jobs: CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 + # We don't want using an msys bash to "hide" all the other already + # installed tools, that would require us to install tools into msys that + # are already available otherwise. + MSYS2_PATH_TYPE: inherit + defaults: run: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' @@ -1086,11 +1091,10 @@ jobs: # MSYS2. It dynamically expands to the correct prefix for the active # shell environment. pacman -S --noconfirm --needed --asdeps \ - git bison flex diffutils \ + bison flex \ ${MINGW_PACKAGE_PREFIX}-ccache \ ${MINGW_PACKAGE_PREFIX}-gcc \ ${MINGW_PACKAGE_PREFIX}-icu \ - ${MINGW_PACKAGE_PREFIX}-libbacktrace \ ${MINGW_PACKAGE_PREFIX}-libxml2 \ ${MINGW_PACKAGE_PREFIX}-libxslt \ ${MINGW_PACKAGE_PREFIX}-lz4 \ -- 2.54.0.450.g9ac3f193c0
>From ec313988704520705aaae85b86767187d2f9f27d Mon Sep 17 00:00:00 2001 From: Andres Freund <[email protected]> Date: Fri, 12 Jun 2026 08:46:00 -0400 Subject: [PATCH v13a 4/9] ci: Make our own msys2 install from scratch, instead of moving Previously we relocated the existing install, for performance reasons. However that has two disadvantages: 1) moving the install and then installing the packages we need is slower than just creating a new install 2) It hardcodes that D: is the fast drive, but that turns out to depend on the type of runner used 3) We were not using any caching and therefore downloaded the same files over and over. We could have added caching in the previous approach too, but it'd have been extra work. I previously prototyped handrolling the install, instead of using msys2/setup-msys2@v2, and that does turn out to be a bit faster, but it doesn't include caching. By the time we add that, the overall complexity seems like it'd be too big. The other alternative would be to generate a downloadable release in the pg-vm-images repo. That'd likely be even faster. Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 64 +++++++++++++++++-------------------- 1 file changed, 29 insertions(+), 35 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index 442052b72e1..794a2600a23 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -1063,48 +1063,42 @@ jobs: defaults: run: - shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + shell: msys2 {0} steps: - *windows_disable_defender_step - *window_setup_hosts_step - *checkout_step - # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to - # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses - # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # The pre-existing msys install is on C:\, often a rather slow system + # disk, whereas the workspace is often on a faster, ephemeral disk. + # Using the pre-existing msys would often increase the total runtime of + # this task by ~15 minutes. # - # This reduces the total runtime of this task by ~15 minutes. - # - # robocopy returns 0-7 on success (with various "files copied" bits - # set) and 8+ on real failure, so we have to translate its exit code. - - name: Relocate MSYS2 to D - shell: pwsh - run: | - robocopy C:\msys64 D:\msys64 /E /NJS /NJH /NFL /NDL /NP - if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } - exit 0 - - - name: Setup MSYS2 - run: | - # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the - # MSYS2. It dynamically expands to the correct prefix for the active - # shell environment. - pacman -S --noconfirm --needed --asdeps \ - bison flex \ - ${MINGW_PACKAGE_PREFIX}-ccache \ - ${MINGW_PACKAGE_PREFIX}-gcc \ - ${MINGW_PACKAGE_PREFIX}-icu \ - ${MINGW_PACKAGE_PREFIX}-libxml2 \ - ${MINGW_PACKAGE_PREFIX}-libxslt \ - ${MINGW_PACKAGE_PREFIX}-lz4 \ - ${MINGW_PACKAGE_PREFIX}-make \ - ${MINGW_PACKAGE_PREFIX}-meson \ - ${MINGW_PACKAGE_PREFIX}-perl \ - ${MINGW_PACKAGE_PREFIX}-pkgconf \ - ${MINGW_PACKAGE_PREFIX}-readline \ - ${MINGW_PACKAGE_PREFIX}-zlib \ - ${MINGW_PACKAGE_PREFIX}-zstd + # Instead we use msys2/setup-msys2 to set up our own install. That's + # faster than moving the install from C:\, and also works when the + # workspace is not on a separate disk. + - name: Install MSYS2 + uses: msys2/setup-msys2@66cd2cce69caa17b53920067426061ca1de3a884 # v2.31.1 + with: + msystem: ${{env.MSYSTEM}} + update: true + location: ${{github.workspace}} + install: >- + bison flex + mingw-w64-ucrt-x86_64-ccache + mingw-w64-ucrt-x86_64-gcc + mingw-w64-ucrt-x86_64-icu + mingw-w64-ucrt-x86_64-libxml2 + mingw-w64-ucrt-x86_64-libxslt + mingw-w64-ucrt-x86_64-lz4 + mingw-w64-ucrt-x86_64-make + mingw-w64-ucrt-x86_64-meson + mingw-w64-ucrt-x86_64-perl + mingw-w64-ucrt-x86_64-pkgconf + mingw-w64-ucrt-x86_64-readline + mingw-w64-ucrt-x86_64-zlib + mingw-w64-ucrt-x86_64-zstd - *nix_sysinfo_step -- 2.54.0.450.g9ac3f193c0
>From 4ca78ad49183ba5899ac489970d51dd61e70e8f0 Mon Sep 17 00:00:00 2001 From: Andres Freund <[email protected]> Date: Fri, 12 Jun 2026 17:06:37 -0400 Subject: [PATCH v13a 5/9] squash-or-drop: Use pacboy for shorter package names --- .github/workflows/pg-ci.yml | 28 +++++++++++++++------------- 1 file changed, 15 insertions(+), 13 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index 794a2600a23..41a807af801 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -1086,19 +1086,21 @@ jobs: location: ${{github.workspace}} install: >- bison flex - mingw-w64-ucrt-x86_64-ccache - mingw-w64-ucrt-x86_64-gcc - mingw-w64-ucrt-x86_64-icu - mingw-w64-ucrt-x86_64-libxml2 - mingw-w64-ucrt-x86_64-libxslt - mingw-w64-ucrt-x86_64-lz4 - mingw-w64-ucrt-x86_64-make - mingw-w64-ucrt-x86_64-meson - mingw-w64-ucrt-x86_64-perl - mingw-w64-ucrt-x86_64-pkgconf - mingw-w64-ucrt-x86_64-readline - mingw-w64-ucrt-x86_64-zlib - mingw-w64-ucrt-x86_64-zstd + # name:p means MINGW_PACKAGE_PREFIX-only + pacboy: >- + ccache:p + gcc:p + icu:p + libxml2:p + libxslt:p + lz4:p + make:p + meson:p + perl:p + pkgconf:p + readline:p + zlib:p + zstd:p - *nix_sysinfo_step -- 2.54.0.450.g9ac3f193c0
>From 707a3f799121b514c99746f04cdda87eaf087acd Mon Sep 17 00:00:00 2001 From: Andres Freund <[email protected]> Date: Thu, 11 Jun 2026 00:01:53 -0400 Subject: [PATCH v13a 6/9] ci: Use optimized build for mingw The test runtime dominates over the compile time on GHA. Note that we just need to remove options, as postgres's default is debugoptimized. Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index 41a807af801..ab7950f6f14 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -1127,7 +1127,7 @@ jobs: run: | meson setup \ ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ - -Ddebug=true -Doptimization=g -Db_pch=true \ + -Db_pch=true \ ${{env.MESON_COMMON_FEATURES}} \ ${{env.MESON_FEATURES}} \ -DTAR=${{env.TAR}} \ -- 2.54.0.450.g9ac3f193c0
>From e9e2172ebe07c94848874f9a793518136af34eb1 Mon Sep 17 00:00:00 2001 From: Andres Freund <[email protected]> Date: Wed, 10 Jun 2026 14:49:28 -0400 Subject: [PATCH v13a 7/9] ci: get build/test concurrency from environment Previously we hardcoded the amount of concurrency in the CI definition. This practice IIRC originated from a) using make, where the concurrency is not sourced from the current system b) early cirrus-ci macos runners, which slowed down a lot with full concurrency. Hardcoding the concurrency is problematic as e.g. private repositories have lower concurrency and larger github runners have more cores. For most of the jobs we can rely on just using meson's and ninja's logic for getting the current core count. For the autoconf we can use the nproc helper. To make this at least somewhat understandable, I added the existing sysinfo step to show the number of cores (and made it a bit easier to understand the information already printed). Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 47 +++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index ab7950f6f14..9384ec0d902 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -40,14 +40,6 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 - # At the moment all jobs use 4vcore runners, and none seems to benefit from - # increasing concurrency further. - BUILD_JOBS: 4 - - # It's possible that some jobs benefit from an increased test concurrency, - # but a default of 4 is a safe bet. Individual jobs can override. - TEST_JOBS: 4 - CCACHE_MAXSIZE: "250M" CCACHE_DIR: ${{ github.workspace }}/ccache_dir @@ -215,10 +207,19 @@ jobs: - &nix_sysinfo_step name: sysinfo run: | - id - uname -a + echo -n "Number of cores: " + nproc 2> /dev/null || sysctl hw.physicalcpu hw.logicalcpu + + echo "id: $(id)" + echo "uname: $(uname -a)" + + echo ::group::ulimit ulimit -a -H && ulimit -a -S + echo ::endgroup:: + + echo ::group::env env + echo ::endgroup:: - name: Parse ci-os-only id: os @@ -352,7 +353,7 @@ jobs: - name: Build shell: *su_postgres_shell run: &ninja_build_cmd | - ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps # Decide if it's worth uploading a new version of the ccache cache. If @@ -405,7 +406,7 @@ jobs: meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 echo ::endgroup:: - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + meson test ${{env.MTEST_ARGS}} --no-suite setup ${{env.MTEST_TARGET}} - &linux_collect_cores_step name: Core backtraces @@ -511,7 +512,7 @@ jobs: - name: Build shell: *su_postgres_shell run: | - make -s -j${BUILD_JOBS} world-bin + make -s -j$(nproc) world-bin - *ccache_decide_save_step - *ccache_save_step @@ -519,7 +520,7 @@ jobs: - name: Test world shell: *su_postgres_shell run: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + make -s ${CHECK} ${CHECKFLAGS} -j$(nproc) - *linux_collect_cores_step - *upload_logs_step @@ -611,7 +612,7 @@ jobs: build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start # Run the tests supporting running against an already running - meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + meson test ${{env.MTEST_ARGS}} --setup running build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop @@ -893,9 +894,19 @@ jobs: - name: Sysinfo run: | + echo Number of cores: + bash -c nproc + + echo codepage: chcp + + echo ::group::systeminfo systeminfo + echo ::endgroup:: + + echo ::group::env set + echo ::endgroup:: # The TAP tests build an initdb template under build/tmp_install and # then `robocopy` it into per-test data directories. Robocopy with the @@ -1198,7 +1209,7 @@ jobs: meson setup -Dwerror=true $CONF build-meson echo "::endgroup::" - ninja -C build-meson --quiet -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build-meson --quiet ${{env.MBUILD_TARGET}} # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) @@ -1266,8 +1277,8 @@ jobs: CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" echo "::endgroup::" - make -s -j${{env.BUILD_JOBS}} clean - make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + make -s -j$(nproc) clean + make -s -j$(nproc) -k ${{env.CHECKFLAGS}} \ headerscheck cpluspluscheck \ EXTRAFLAGS='-fmax-errors=10' -- 2.54.0.450.g9ac3f193c0
>From 9ebb45746bbb04b1f44692f0f00aed9cfd67d3dd Mon Sep 17 00:00:00 2001 From: Andres Freund <[email protected]> Date: Wed, 10 Jun 2026 13:09:10 -0400 Subject: [PATCH v13a 8/9] ci: Make runs-on overridable Previously what runners CI ran on was hardcoded. Unfortunately that makes it harder to use faster or self hosted runners (e.g. in a private repository where the runners are very slow, or self hosted runners for cfbot). Fix that by allowing to override the runner types via repository/organization variables. How to do that is documented in src/tools/ci/README. Testing faster github hosted runners showed that we have some dependencies on specific locations, fix most of those. The remainder will be in a separate commit, changing how we deal with the msys2 installation. Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 60 ++++++++++++++++++++++++------------- src/tools/ci/README | 21 +++++++++++++ 2 files changed, 60 insertions(+), 21 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index 9384ec0d902..c737173abe0 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -106,12 +106,6 @@ env: --with-uuid=ossp --with-zstd - # Centrally define the version of linux runners, to make it easier to - # update. We don't just want to use ubuntu-latest, as it's not implausible - # there will be breakage when that switches to the next ubuntu version. - _LINUX_RUNS_ON: &linux_runs_on | - ubuntu-24.04 - # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/'. CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/main @@ -154,7 +148,8 @@ jobs: warn-if-not-opted-in: name: Report if not opted into CI if: ${{vars.PG_CI_ENABLED != '1'}} - runs-on: ubuntu-slim + # See the setup task for an explanation + runs-on: ${{ case(vars.pg_ci_runs_on_setup != '', vars.pg_ci_runs_on_setup, 'ubuntu-slim') }} steps: - name: Warn env: @@ -184,21 +179,43 @@ jobs: # Only run CI if repo owner opted in. If this task is skipped due to the # if, none of it's depending tasks (i.e. the actual CI tasks) run either. if: ${{vars.PG_CI_ENABLED == '1'}} - runs-on: *linux_runs_on timeout-minutes: 1 outputs: + # Are certain tasks enabled? linux: ${{ steps.os.outputs.linux }} macos: ${{ steps.os.outputs.macos }} windows: ${{ steps.os.outputs.windows }} mingw: ${{ steps.os.outputs.mingw }} compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} sanitycheck: ${{ steps.os.outputs.sanitycheck }} + + # What runners to run on: + # + # To allow over-riding what runners jobs use, we allow setting + # repository / org level variables influencing that choice on a + # per-host-operating-system basis. + # + # This also makes it easier to change the version of linux / windows + # used centrally. We don't just want to use ubuntu-latest, as it's not + # implausible there will be breakage when that switches to the next + # ubuntu version. + runs_on_linux: ${{ case(vars.PG_CI_RUNS_ON_LINUX != '', vars.PG_CI_RUNS_ON_LINUX, 'ubuntu-24.04') }} + runs_on_windows: ${{ case(vars.PG_CI_RUNS_ON_WINDOWS != '', vars.PG_CI_RUNS_ON_WINDOWS, 'windows-2022') }} + runs_on_macos: ${{ case(vars.PG_CI_RUNS_ON_MACOS != '', vars.PG_CI_RUNS_ON_MACOS, 'macos-15') }} + # runs_on_setup: an output can't be used as the input of the setup job + # itself, so it's done inline below. + + # Exports: + # # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} + # Can't use the output from above, so repeated verbatim here + runs-on: ${{ case(vars.pg_ci_runs_on_setup != '', vars.pg_ci_runs_on_setup, 'ubuntu-slim') }} + steps: # Anchor reused by other jobs further down. GitHub Actions supports YAML # anchors/aliases but not merge keys, so the alias copies the whole step @@ -258,7 +275,7 @@ jobs: if: | !cancelled() && needs.setup.outputs.sanitycheck == 'true' - runs-on: *linux_runs_on + runs-on: ${{ needs.setup.outputs.runs_on_linux }} timeout-minutes: 15 container: &linux_ci_container image: ${{ needs.setup.outputs.container_linux_ci }} @@ -446,7 +463,7 @@ jobs: !cancelled() && needs.setup.outputs.linux == 'true' && needs.sanity-check.result != 'failure' - runs-on: *linux_runs_on + runs-on: ${{ needs.setup.outputs.runs_on_linux }} container: *linux_ci_container timeout-minutes: 60 @@ -540,7 +557,7 @@ jobs: name: Linux - Meson (32-bit) needs: [setup, sanity-check] if: *linux_job_if - runs-on: *linux_runs_on + runs-on: ${{ needs.setup.outputs.runs_on_linux }} container: *linux_ci_container timeout-minutes: 60 env: *linux_env @@ -632,7 +649,7 @@ jobs: name: Linux - Meson (64-bit) needs: [setup, sanity-check] if: *linux_job_if - runs-on: *linux_runs_on + runs-on: ${{ needs.setup.outputs.runs_on_linux }} container: *linux_ci_container timeout-minutes: 60 env: *linux_env @@ -688,7 +705,7 @@ jobs: !cancelled() && needs.setup.outputs.macos == 'true' && needs.sanity-check.result != 'failure' - runs-on: macos-15 + runs-on: ${{ needs.setup.outputs.runs_on_macos }} timeout-minutes: 60 env: MACPORTS_CACHE: ${{ github.workspace }}/macports-cache @@ -845,7 +862,7 @@ jobs: !cancelled() && needs.setup.outputs.windows == 'true' && needs.sanity-check.result != 'failure' - runs-on: windows-2022 + runs-on: ${{ needs.setup.outputs.runs_on_windows }} timeout-minutes: 60 # As described at the top of the task, split the tests across two runners @@ -861,7 +878,7 @@ jobs: env: # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 - PG_REGRESS_SOCK_DIR: 'd:\pgsock' + PG_REGRESS_SOCK_DIR: ${{ github.workspace }}/pgsock TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- @@ -967,7 +984,8 @@ jobs: - name: Install dependencies shell: pwsh run: | - # meson is not preinstalled on windows-2022. Install via pip + # meson is not preinstalled, at least on windows-2022. Install via + # pip echo ::group::pip python -m pip install --upgrade meson if (!$?) { throw 'cmdfail' } @@ -999,7 +1017,7 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} + run: mkdir "${{env.PG_REGRESS_SOCK_DIR}}" - &windows_setup_debugger_step name: Setup Windows debugger @@ -1046,12 +1064,12 @@ jobs: !cancelled() && needs.setup.outputs.mingw == 'true' && needs.sanity-check.result != 'failure' - runs-on: windows-2022 + runs-on: ${{ needs.setup.outputs.runs_on_windows }} timeout-minutes: 60 env: # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 - PG_REGRESS_SOCK_DIR: 'd:\pgsock' + PG_REGRESS_SOCK_DIR: ${{ github.workspace }}/pgsock TAR: "c:/windows/system32/tar.exe" MSYS: winjitdebug @@ -1127,7 +1145,7 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} + run: mkdir "${{env.PG_REGRESS_SOCK_DIR}}" - *windows_setup_debugger_step @@ -1173,7 +1191,7 @@ jobs: !cancelled() && needs.setup.outputs.compilerwarnings == 'true' && needs.sanity-check.result != 'failure' - runs-on: *linux_runs_on + runs-on: ${{ needs.setup.outputs.runs_on_linux }} timeout-minutes: 60 container: image: ${{ needs.setup.outputs.container_linux_ci_docs }} diff --git a/src/tools/ci/README b/src/tools/ci/README index 642e518c296..9276d4a654c 100644 --- a/src/tools/ci/README +++ b/src/tools/ci/README @@ -75,3 +75,24 @@ messages. Currently the following controls are available: Only runs CI on operating systems specified. This can be useful when addressing portability issues affecting only a subset of platforms. + + +Controlling which runners CI uses +================================= + +By default each job runs on a GitHub-hosted runner appropriate for its +operating system. This can be overridden on a per-operating-system basis by +creating repository or organization variables, at +https://github.com/<username>/<reponame>/settings/variables/actions + +This is useful to run on faster GitHub-hosted runners, or on self-hosted +runners. + +The following variables are recognized. If a variable is unset or empty, a +default runner suitable for that operating system is used: + +- PG_CI_RUNS_ON_LINUX: Linux jobs, including SanityCheck and CompilerWarnings +- PG_CI_RUNS_ON_WINDOWS: the Windows (VS) and MinGW jobs +- PG_CI_RUNS_ON_MACOS: the macOS job +- PG_CI_RUNS_ON_SETUP: the lightweight bookkeeping jobs (opt-in warning and + job setup) -- 2.54.0.450.g9ac3f193c0
>From b5412dc3b24334f8eb6396b1e37c05a7096eb03e Mon Sep 17 00:00:00 2001 From: Andres Freund <[email protected]> Date: Fri, 12 Jun 2026 12:57:36 -0400 Subject: [PATCH v13a 9/9] ci: Move to per-job naming for containers Previously we only had two types of containers, "ci" for everything but the CompilerWarnings job and ci_docs that had the tools necessary to build the docs as part of CompilerWarnings. But that was unnecessarily heavy for several of the jobs. As the pull time is a noticeable performance factor, that's not great. Instead use containers that are named by their job. Today some of those are the same, but after this change we can optimize the containers for their jobs without a problem. It probably would make sense to allow over-riding CONTAINER_REPO on a per-repo/org basis, but that seems better done as a separate change. Discussion: https://postgr.es/m/a2ejn7lfqolutzz7kozalbhy3bixdrujb4buc3pgbtlk4am2ba@wbv6v7riia33 --- .github/workflows/pg-ci.yml | 32 +++++++++++++++++++++----------- 1 file changed, 21 insertions(+), 11 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index c737173abe0..4034f76e0a1 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -108,9 +108,12 @@ env: # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/'. + # + # The containers are named + # ghcr.io/anarazel/pg-vm-images/<branch>/linux_debian_ci/<debian version>/<ci-job>:latest CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/main - CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest - CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest + CONTAINER_LINUX_CI_NAME: linux_debian_ci/trixie + CONTAINER_TAG: latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. @@ -193,7 +196,7 @@ jobs: # # To allow over-riding what runners jobs use, we allow setting # repository / org level variables influencing that choice on a - # per-host-operating-system basis. + # per-host-operating-system basis. See also src/tools/ci/README # # This also makes it easier to change the version of linux / windows # used centrally. We don't just want to use ubuntu-latest, as it's not @@ -210,8 +213,8 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} - container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} + container_linux_ci_base: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_NAME }} + container_tag: ${{ env.CONTAINER_TAG }} # Can't use the output from above, so repeated verbatim here runs-on: ${{ case(vars.pg_ci_runs_on_setup != '', vars.pg_ci_runs_on_setup, 'ubuntu-slim') }} @@ -277,8 +280,8 @@ jobs: needs.setup.outputs.sanitycheck == 'true' runs-on: ${{ needs.setup.outputs.runs_on_linux }} timeout-minutes: 15 - container: &linux_ci_container - image: ${{ needs.setup.outputs.container_linux_ci }} + container: + image: ${{ needs.setup.outputs.container_linux_ci_base }}/sanity-check:${{ needs.setup.outputs.container_tag }} # Options passed to all linux containers. Not all of the jobs need # all of them, but it's easier to just define them centrally. @@ -464,7 +467,9 @@ jobs: needs.setup.outputs.linux == 'true' && needs.sanity-check.result != 'failure' runs-on: ${{ needs.setup.outputs.runs_on_linux }} - container: *linux_ci_container + container: + image: ${{ needs.setup.outputs.container_linux_ci_base }}/linux-autoconf:${{ needs.setup.outputs.container_tag }} + options: *linux_container_options timeout-minutes: 60 env: &linux_env @@ -558,7 +563,9 @@ jobs: needs: [setup, sanity-check] if: *linux_job_if runs-on: ${{ needs.setup.outputs.runs_on_linux }} - container: *linux_ci_container + container: + image: ${{ needs.setup.outputs.container_linux_ci_base }}/linux-meson-32:${{ needs.setup.outputs.container_tag }} + options: *linux_container_options timeout-minutes: 60 env: *linux_env @@ -650,7 +657,9 @@ jobs: needs: [setup, sanity-check] if: *linux_job_if runs-on: ${{ needs.setup.outputs.runs_on_linux }} - container: *linux_ci_container + container: + image: ${{ needs.setup.outputs.container_linux_ci_base }}/linux-meson-64:${{ needs.setup.outputs.container_tag }} + options: *linux_container_options timeout-minutes: 60 env: *linux_env @@ -1194,7 +1203,8 @@ jobs: runs-on: ${{ needs.setup.outputs.runs_on_linux }} timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.container_linux_ci_docs }} + image: ${{ needs.setup.outputs.container_linux_ci_base }}/compiler-warnings:${{ needs.setup.outputs.container_tag }} + options: *linux_container_options env: # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. -- 2.54.0.450.g9ac3f193c0
