Hi, One more revision:
Attached are the prior patches plus one incremental one (to be squashed), making the macports caching a bit smarter. The performance improvements after touching src/tools/ci/ci_macports_packages.sh, updating the package list, or starting with a clean cache and then failing during build/test seems clearly worth the complexity. Greetings, Andres
>From 81ed04c725db8e575eaa9043ca4aadab699f6e8f Mon Sep 17 00:00:00 2001 From: Andres Freund <[email protected]> Date: Wed, 3 Jun 2026 22:45:31 -0400 Subject: [PATCH v12a 1/3] ci: Add GitHub Actions based CI Cirrus CI, which the project used for CI until now, has shut down on June 1, 2026. Replace it with GitHub Actions. GitHub Actions was selected because it has unlimited runner time for public repositories. The GitHub Actions based CI currently covers: - SanityCheck - Linux - Autoconf - Linux - Meson, (32-bit and 64-bit) - macOS - Meson - Windows (Visual Studio + Meson and MinGW + Meson) - CompilerWarnings BSD coverage is left for later, as it requires more work. Note that, for performance reasons, use of address sanitizer was moved to the Linux - Meson (64-bit) task. While Actions workflows in new forks are disabled by default, existing forks that pull new changes into the repository will automatically start running CI. That may not be desired. There however is no way native to Actions to prevent this. To avoid that, each repository that wants real CI to run needs to explicitly opt into doing so, by creating the 'PG_CI_ENABLED' repository variable with the value 1. To make that less confusing, emit a summary whenever we skip running CI, with a message explaining how to enable CI. The remaining cirrus-ci support will be removed in a subsequent commit, to make review easier. Back-branches will be updated later, after being sure that workflow runs correctly on master. Author: Nazir Bilal Yavuz <[email protected]> Author: Andres Freund <[email protected]> Author: Jelte Fennema-Nio <[email protected]> Reviewed-by: Jacob Champion <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Zsolt Parragi <[email protected]> Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- .github/workflows/pg-ci.yml | 1199 ++++++++++++++++++++++++++ src/tools/ci/README | 90 +- src/tools/ci/ci_macports_packages.sh | 22 +- 3 files changed, 1253 insertions(+), 58 deletions(-) create mode 100644 .github/workflows/pg-ci.yml diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml new file mode 100644 index 00000000000..dea8d95d729 --- /dev/null +++ b/.github/workflows/pg-ci.yml @@ -0,0 +1,1199 @@ +# GitHub Actions CI configuration for PostgreSQL +# +# For instructions on how to enable / disable CI integration in a repository +# and further details, see src/tools/ci/README +# +# https://docs.github.com/en/actions/reference/workflows-and-actions/workflow-syntax +# is a good starting point for documentation about GitHub Actions. + +name: CI for PostgreSQL + +on: + push: + # TODO: It might make sense to also add PR based triggers, to make it easier + # to use PRs on one's own repo, but it's a tad more complicated than just + # adding the 'pull_request' event, as naively doing so would often lead to + # running CI twice. + +# Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo +# contents during checkout. +permissions: + contents: read + +concurrency: + # For anything other than stable branches, we want there to only be one + # workflow active for that branch. But on stable branches & master, we + # neither want to wait for prior runs, nor to cancel them, so that each + # separately pushed commit is tested. We achieve that by setting a unique + # concurrency group when on such a branch. + group: | + ${{github.workflow }}-${{ + case(github.ref == 'refs/heads/master' || + (startsWith(github.ref, 'refs/heads/REL_') && endsWith(github.ref, '_STABLE')), + github.run_id, + github.ref) + }} + cancel-in-progress: true + +env: + # The lower depth accelerates git clone. Use a bit of depth so that + # concurrent jobs and retrying older runs have a chance of working. + CLONE_DEPTH: 500 + + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir + + # Check target for the autoconf builds. Can be set to e.g. check to only + # test the main regression tests. + CHECK: check-world PROVE_FLAGS=--timer + CHECKFLAGS: -Otarget + + # Build test dependencies as part of the build step, to see compiler + # errors/warnings in one place. + MBUILD_TARGET: all testprep + MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests + TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth + + # Postgres config args for the meson builds, shared between all meson tasks + # except the 'SanityCheck' task + MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true + + # Meson feature flags shared by all meson tasks, except: + # SanityCheck: uses almost no dependencies. + # Windows - VS: has fewer dependencies than listed here, so defines its own. + # Linux: uses the 'auto' feature option to test meson feature autodetection. + MESON_COMMON_FEATURES: >- + -Dauto_features=disabled + -Ddocs=enabled + -Dicu=enabled + -Dldap=enabled + -Dlibxml=enabled + -Dlibxslt=enabled + -Dlz4=enabled + -Dplperl=enabled + -Dplpython=enabled + -Dpltcl=enabled + -Dreadline=enabled + -Dssl=openssl + -Dtap_tests=enabled + -Dzlib=enabled + -Dzstd=enabled + + # Shared between the Linux autoconf job and the CompilerWarnings jobs + LINUX_CONFIGURE_FEATURES: >- + --with-gssapi + --with-icu + --with-ldap + --with-libcurl + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=ossp + --with-zstd + + # Centrally define the version of linux runners, to make it easier to + # update. We don't just want to use ubuntu-latest, as it's not implausible + # there will be breakage when that switches to the next ubuntu version. + _LINUX_RUNS_ON: &linux_runs_on | + ubuntu-24.04 + + # Debian Trixie containers used by all Linux jobs. Built by + # 'https://github.com/anarazel/pg-vm-images/'. + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest + + # The full set of OS / job selectors recognized by the `ci-os-only:` + # commit-message directive parsed in the `setup` job below. + CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" + + +jobs: + + # Job: Report if repository has not opted into CI + # + # Do not run CI unless the repository owner opts in, to avoid resource waste + # in all the forks of postgres (new forks have workflows disabled by + # default, but old ones don't). Unfortunately there's no declarative way to + # do so. + # + # To make the lack of actual CI due to missing opt-in more visible, emit a + # summary explaining how CI can be opted into and how the entire workflow, + # including this warning, can be disabled. + warn-if-not-opted-in: + name: Report if not opted into CI + if: ${{vars.PG_CI_ENABLED != '1'}} + runs-on: ubuntu-slim + steps: + - name: Warn + env: + MSG: | + > [!IMPORTANT] + > ${{github.workflow}} has not been opted into in this repository + > + > To opt into ${{github.workflow}}, go to + > ${{github.server_url}}/${{github.repository}}/settings/variables/actions + > and create a new repository variable named PG_CI_ENABLED, with + > the value 1. + > + > To avoid seeing this message over and over, go to + > ${{github.server_url}}/${{github.repository}}/actions/workflows/pg-ci.yml + > and click on the three dots at the top right and choose + > "Disable workflow" + run: | + echo "$MSG" |tee -a "$GITHUB_STEP_SUMMARY" + + + # Job: Determine enabled jobs + # + # Parses "ci-os-only: ..." from the commit message and exposes flags + # consumed by the jobs' `if:` conditions. + setup: + name: Determine enabled jobs + # Only run CI if repo owner opted in. If this task is skipped due to the + # if, none of it's depending tasks (i.e. the actual CI tasks) run either. + if: ${{vars.PG_CI_ENABLED == '1'}} + runs-on: *linux_runs_on + timeout-minutes: 1 + outputs: + linux: ${{ steps.os.outputs.linux }} + macos: ${{ steps.os.outputs.macos }} + windows: ${{ steps.os.outputs.windows }} + mingw: ${{ steps.os.outputs.mingw }} + compilerwarnings: ${{ steps.os.outputs.compilerwarnings }} + sanitycheck: ${{ steps.os.outputs.sanitycheck }} + # Re-export workflow-level env vars that other jobs need to reference + # from contexts (e.g. `jobs.<id>.container.image`) where the `env` + # context is not available. + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} + + steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + + - name: Parse ci-os-only + id: os + env: + MSG: ${{ github.event.head_commit.message }} + shell: bash + run: | + all_os=${CI_OS_ONLY_JOBS} + if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then + sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) + echo "ci-os-only selection: $sel" + else + sel="$all_os" + fi + for o in $all_os; do + if echo " $sel " | grep -qE "[ ,]$o[ ,]"; then + echo "$o=true" >> "$GITHUB_OUTPUT" + else + echo "$o=false" >> "$GITHUB_OUTPUT" + fi + done + cat "$GITHUB_OUTPUT" + + + # Job: SanityCheck + # + # To avoid unnecessarily spinning up a lot of VMs / containers for entirely + # broken commits, have a minimal task that all others depend on. + # + # SPECIAL: + # - Builds with --auto-features=disabled and thus almost no enabled + # dependencies + sanity-check: + name: SanityCheck + needs: setup + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' + runs-on: *linux_runs_on + timeout-minutes: 15 + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # + # --privileged is needed so the prepare step can write to sysctls + # under /proc/sys (it's mounted read-only without it). We use it to + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # + # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, + # kill9's, and restarts postgres; with the container's small PID + # space a new postgres can recycle the dead postmaster's PID before + # pg_ctl's postmaster.pid check notices, producing spurious "node X + # is already running" failures. SysV shm in the test also relies on + # host-like IPC behavior. + # + # --ulimit raises memlock and core dump size. Memlock is needed for + # running the AIO tests. + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 + env: + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" + + steps: + - *nix_sysinfo_step + + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} + restore-keys: | + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- + + - &linux_prepare_workspace_step + name: Prepare workspace + run: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + # This is only needed for some of the tasks using this, but it + # doesn't harm to have this enabled. + sysctl -w kernel.io_uring_disabled=0 + + cat >> /etc/hosts <<-EOF + 127.0.0.1 pg-loadbalancetest + 127.0.0.2 pg-loadbalancetest + 127.0.0.3 pg-loadbalancetest + EOF + + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... + - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" + run: | + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: &ninja_build_cmd | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps + + # TODO: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} + + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup || exit 1 + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --no-suite setup ${{env.MTEST_TARGET}} + + - &linux_collect_cores_step + name: Core backtraces + if: failure() && !cancelled() + run: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() + uses: actions/upload-artifact@v7 + with: + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt + build/meson-logs/** + **/config.log + if-no-files-found: ignore + + + # Job: Linux - Autoconf + # + # SPECIAL: + # - Uses undefined & alignment sanitizers (sanitizer failures are typically + # printed in the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + # - Uses postgres specific CPPFLAGS that increase test coverage + # - Enables --link for pg_upgrade + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: *linux_runs_on + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and ubuntu, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + CPPFLAGS: -DRELCACHE_FORCE_RELEASE -DENFORCE_REGRESSION_TEST_NAME_RESTRICTIONS + PG_TEST_PG_UPGRADE_MODE: --link + run: &linux_update_config_cmd | + echo "CPPFLAGS=$CPPFLAGS" >> "$GITHUB_ENV" + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_PG_UPGRADE_MODE=${PG_TEST_PG_UPGRADE_MODE}" >> "$GITHUB_ENV" + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace_step + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores_step + - *upload_logs_step + + + # Job: Linux - Meson (32-bit) + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: *linux_runs_on + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace_step + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_cmd + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + # Test running against existing PG instance. + # + # linux-meson-32 chosen because it's currently comparatively fast + - name: Test running + shell: *su_postgres_shell + run: | + ulimit -c unlimited + + # Ensure install exists, in case somebody is debugging a failing + # test and reorders this to be before "Test world". + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + # Make libraries discoverable (the x86_64 reference is a meson + # oddity) + export LD_LIBRARY_PATH="$(pwd)/build/tmp_install/usr/local/pgsql/lib/x86_64-linux-gnu/:$LD_LIBRARY_PATH" + + build/tmp_install/usr/local/pgsql/bin/initdb -N build/runningcheck --no-instructions -A trust + echo "include '$(pwd)/src/tools/ci/pg_ci_base.conf'" >> build/runningcheck/postgresql.conf + + # Log into a place that will be archived in case of failure + mkdir -p build/testrun + build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start + + # Run the tests supporting running against an already running + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} --setup running + + build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop + + - *linux_collect_cores_step + - *upload_logs_step + + + # Linux - Meson (64-bit) + # + # SPECIAL: + # - Uses address sanitizer, (sanitizer failures are typically printed in the + # server log). We test asan with meson rather than autoconf, as it's a bit + # faster at running the tests. + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: *linux_runs_on + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace_step + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_cmd + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores_step + - *upload_logs_step + + + # Job: macOS - Meson + # + # SPECIAL: + # - Enables --clone for pg_upgrade and pg_combinebackup + # - Specifies configuration options that test reading/writing/copying of node trees + # - Specifies debug_parallel_query=regress, to catch related issues during CI + macos: + name: macOS - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.macos == 'true' && + needs.sanity-check.result != 'failure' + runs-on: macos-15 + timeout-minutes: 60 + env: + MACPORTS_CACHE: ${{ github.workspace }}/macports-cache + + MESON_FEATURES: >- + -Dbonjour=enabled + -Ddtrace=enabled + -Dgssapi=enabled + -Dlibcurl=enabled + -Dnls=enabled + -Duuid=e2fs + + MACOS_PACKAGE_LIST: >- + ccache + icu + kerberos5 + lz4 + meson + openldap + openssl + p5.34-io-tty + p5.34-ipc-run + python312 + tcl + zstd + + CC: ccache cc + CXX: ccache c++ + CFLAGS: -Og -ggdb + CXXFLAGS: -Og -ggdb + PG_TEST_PG_UPGRADE_MODE: --clone + PG_TEST_PG_COMBINEBACKUP_MODE: --clone + + # Several buildfarm animals enable these options. Without testing them + # during CI, it would be easy to cause breakage on the buildfarm with CI + # passing. + PG_TEST_INITDB_EXTRA_OPTS: >- + -c debug_copy_parse_plan_trees=on + -c debug_write_read_parse_plan_trees=on + -c debug_raw_expression_coverage_test=on + -c debug_parallel_query=regress + + steps: + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + + - name: Setup core files + run: | + mkdir -p $HOME/cores + sudo sysctl kern.corefile="$HOME/cores/core.%P" + + - name: "Macports: Compute cache key" + id: mpkey + run: | + macos_major=$(sw_vers -productVersion | sed 's/\..*//') + pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) + script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" + + - name: "MacPorts: Restore cache" + uses: actions/cache@v5 + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mpkey.outputs.key }} + + # Use MacPorts, even though Homebrew is installed. The installation + # of the additional packages we need would take quite a while with + # Homebrew, even if we cache the downloads. We can't cache all of + # Homebrew, because it's already large. So we use MacPorts. To cache + # the installation we create a .dmg file that we mount if it already + # exists. + # XXX: The reason for the direct p5.34* references is that we'd need + # the large MacPort tree around to figure out that p5-io-tty is + # actually p5.34-io-tty. Using the unversioned name works, but + # updates MacPorts every time. + - name: "MacPorts: Install dependencies" + env: + # Pass token so the script's GitHub API call to list MacPorts + # releases isn't subject to the 60/h/IP unauthenticated rate + # limit (shared across all jobs on the runner's IP). + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST + # system python doesn't provide headers + sudo /opt/local/bin/port select python3 python312 + # Make macports install visible to subsequent steps + echo /opt/local/sbin >> "$GITHUB_PATH" + echo /opt/local/bin >> "$GITHUB_PATH" + + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ + run: | + meson setup \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ + --buildtype=debug \ + -Dextra_include_dirs=/opt/local/include \ + -Dextra_lib_dirs=/opt/local/lib \ + -Ddarwin_sysroot=none \ + ${MESON_COMMON_FEATURES} \ + ${MESON_FEATURES} \ + build + + - name: Build + run: *ninja_build_cmd + + - *ccache_save_step + + - name: Test world + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd + + - name: Core backtraces + if: failure() && !cancelled() + run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" + + - *upload_logs_step + + + # Job: Windows - Visual Studio + # + # If we were to execute tests in this job serially, this would be the + # slowest job by a good margin. To avoid that, use a matrix in combination + # with meson test's --slice SLICE/NUM_SLICES mechanism to split the tests + # across two runners. + windows-vs: + name: Windows - Visual Studio - Slice ${{ matrix.slice}}/${{ matrix.num_slices}} + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.windows == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + + # As described at the top of the task, split the tests across two runners + # for performance. The gains from additional concurrency diminish + # relatively quickly, due to each instance having to install dependencies + # and build postgres. + strategy: + fail-fast: false + matrix: + num_slices: [2] + slice: [1, 2] + + env: + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MESON_FEATURES: >- + -Dauto_features=disabled + -Dcpp_args=/std:c++20 + -Dldap=enabled + -Dplperl=enabled + -Dplpython=enabled + -Dssl=openssl + -Dtap_tests=enabled + + defaults: + run: + shell: cmd + + steps: + - &windows_disable_defender_step + name: Disable Windows Defender + shell: pwsh + run: | + Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable + # Verify Defender status + $status = Get-MpComputerStatus -ErrorAction SilentlyContinue + if ($status) { + Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" + Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" + } + + - *checkout_step + + - name: Sysinfo + run: | + chcp + systeminfo + set + + # The TAP tests build an initdb template under build/tmp_install and + # then `robocopy` it into per-test data directories. Robocopy with the + # default /COPY:DAT flag doesn't copy ACLs — destinations inherit from + # their parent dir. On GitHub-hosted Windows runners the workspace's + # inherited ACL grants Administrators:(F) and Users:(RX) but does NOT + # grant the runner user (runneradmin) directly. That matters because + # pg_ctl on Windows uses CreateRestrictedProcess to drop admin + # privileges from postmaster, so the postmaster process has the user + # SID in its token but no longer the Administrators group — leaving it + # with only "Users:(RX)" on pg_control and friends, which causes + # "PANIC: could not open file global/pg_control: Permission denied". + # + # Fix it once on the workspace dir with (OI)(CI) inheritance flags so + # every file/dir created underneath gets an explicit grant for the + # current user. + - name: Grant workspace ACL to runner user + shell: pwsh + run: | + icacls "${{ github.workspace }}" /grant "${env:USERNAME}:(OI)(CI)F" /Q | Out-Null + Write-Host "Granted Full Control to $env:USERNAME on ${{ github.workspace }}" + + # postgres' plpython3u loads python3.dll (the stable-ABI forwarder) + # which in turn loads whichever python3NN.dll the Windows loader finds + # first on PATH. On windows-2022 `C:\Program Files\Mercurial\` ships + # its own python3.dll + python39.dll and appears on PATH *before* the + # hostedtoolcache Python 3.12 — so without intervention the backend + # ends up running Python 3.9 while postgres' stdlib search uses 3.12, + # producing `ImportError: cannot import name 'text_encoding' from + # 'io'` (the 3.12 `io.py` calling into 3.9's `_io`). + # + # Drop Mercurial's directory from PATH so the hostedtoolcache + # python3.dll wins the DLL search. + - name: Remove Mercurial from PATH + shell: pwsh + run: | + $filtered = ($env:PATH -split ';' | + Where-Object { $_ -and ($_ -notmatch '\\Mercurial\\?$') }) -join ';' + Add-Content $env:GITHUB_ENV "PATH=$filtered" + Write-Host "Removed Mercurial entries from PATH" + + # Install some dependencies via msys64, that seems to be the fastest and + # most reliable + - name: Install dependencies, Mingw + shell: 'C:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + run: | + # Install some dependencies via msys64, that seems to be the fastest + # and most reliable + pacman -S --noconfirm --needed --asdeps \ + bison flex + + # Make bison and flex visible + echo C:/msys64/usr/bin >> "$GITHUB_PATH" + + # Don't prefer mingw's perl + echo C:/Strawberry/perl/bin >> "$GITHUB_PATH" + + - name: Install dependencies + shell: pwsh + run: | + # meson is not preinstalled on windows-2022. Install via pip + echo ::group::pip + python -m pip install --upgrade meson + if (!$?) { throw 'cmdfail' } + echo ::endgroup:: + + # Install IPC::Run. + # - recommends_policy=0 keeps cpan from pulling in IO::Tty / IO::Pty, + # which don't build on Windows ("This module requires a POSIX + # compliant system to work"). + # - Pin to NJM/IPC-Run-20250809.0 because TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (changed pipe stdio + # handling). See upstream pg-vm-images commit ff5238afa3 and + # the thread at + # https://postgr.es/m/CAN55FZ06xanSbJdHe-CurjX_qNuBWZDEvS1kAk36L38YCtZXnw%40mail.gmail.com + echo ::group::cpan_ipc_run + "o conf recommends_policy 0`no conf commit`nnotest install NJM/IPC-Run-20250809.0.tar.gz" | cpan + if (!$?) { throw 'cmdfail' } + perl -mIPC::Run -e 1 + if (!$?) { throw 'cmdfail' } + echo ::endgroup:: + + - &window_setup_hosts_step + name: Setup hosts file + shell: pwsh + run: | + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.1 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.2 pg-loadbalancetest" + Add-Content c:\Windows\System32\Drivers\etc\hosts "127.0.0.3 pg-loadbalancetest" + + - name: Setup socket directory + shell: cmd + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} + + - name: Configure + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -DTAR=${{env.TAR}} ^ + build + + - name: Build + run: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + ninja -C build ${{env.MBUILD_TARGET}} || exit 1 + ninja -C build -t missingdeps + + - name: Test world + env: + # As described at the top of the task, split the tests across two + # runners for performance. It's not the prettiest to implement this + # by prepending to MTEST_TARGET, but a more complicated solution + # doesn't seem worth it. + MTEST_TARGET: --slice ${{ matrix.slice}}/${{ matrix.num_slices}} ${{env.MTEST_TARGET}} + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd + + # TODO: We need to collect crashlogs but for them to be generated, we'd + # have to configure the JIT Debugger to do so. cdb.exe is installed on + # the runner so that is possible. + - *upload_logs_step + + + # Job: Windows - MinGW - Meson + windows-mingw: + name: Windows - MinGW - Meson + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.mingw == 'true' && + needs.sanity-check.result != 'failure' + runs-on: windows-2022 + timeout-minutes: 60 + env: + # Avoid port conflicts between concurrent tap tests + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" + + MSYS: winjitdebug + CHERE_INVOKING: 1 + MSYSTEM: UCRT64 + + # Keep -Dnls explicitly disabled, as the number of files it creates + # causes a noticeable slowdown. + MESON_FEATURES: >- + -Dnls=disabled + + CCACHE_MAXSIZE: "500M" + CCACHE_SLOPPINESS: pch_defines,time_macros + CCACHE_DEPEND: 1 + + defaults: + run: + shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' + + steps: + - *windows_disable_defender_step + - *window_setup_hosts_step + - *checkout_step + + # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to + # D:\ (faster ephemeral data disk). Every subsequent MSYS2 step uses + # D:\msys64\usr\bin\bash.exe via the job's `defaults.run.shell`. + # + # This reduces the total runtime of this task by ~15 minutes. + # + # robocopy returns 0-7 on success (with various "files copied" bits + # set) and 8+ on real failure, so we have to translate its exit code. + - name: Relocate MSYS2 to D + shell: pwsh + run: | + robocopy C:\msys64 D:\msys64 /E /MT:16 /NJS /NJH /NFL /NDL /NP + if ($LASTEXITCODE -ge 8) { exit $LASTEXITCODE } + exit 0 + + - name: Setup MSYS2 + run: | + # ${MINGW_PACKAGE_PREFIX} is an environment variable used in the + # MSYS2. It dynamically expands to the correct prefix for the active + # shell environment. + pacman -S --noconfirm --needed --asdeps \ + git bison flex diffutils \ + ${MINGW_PACKAGE_PREFIX}-ccache \ + ${MINGW_PACKAGE_PREFIX}-gcc \ + ${MINGW_PACKAGE_PREFIX}-icu \ + ${MINGW_PACKAGE_PREFIX}-libbacktrace \ + ${MINGW_PACKAGE_PREFIX}-libxml2 \ + ${MINGW_PACKAGE_PREFIX}-libxslt \ + ${MINGW_PACKAGE_PREFIX}-lz4 \ + ${MINGW_PACKAGE_PREFIX}-make \ + ${MINGW_PACKAGE_PREFIX}-meson \ + ${MINGW_PACKAGE_PREFIX}-perl \ + ${MINGW_PACKAGE_PREFIX}-pkgconf \ + ${MINGW_PACKAGE_PREFIX}-readline \ + ${MINGW_PACKAGE_PREFIX}-zlib \ + ${MINGW_PACKAGE_PREFIX}-zstd + + - *nix_sysinfo_step + + - name: Install additional dependencies + run: | + # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 + # broke postgres tap tests on Windows (pipe stdio handling). + # See pg-vm-images commit ff5238afa3. + echo ::group::cpan_ipc_run + (echo; echo o conf recommends_policy 0; echo notest install NJM/IPC-Run-20250809.0.tar.gz) | cpan + perl -mIPC::Run -e 1 + echo ::endgroup:: + + - name: Setup socket directory + shell: cmd + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} + + - *ccache_restore_step + + - name: Configure + run: | + meson setup \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ + -Ddebug=true -Doptimization=g -Db_pch=true \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ + build + + - name: Build + run: *ninja_build_cmd + + - *ccache_save_step + + - name: Test world + run: *meson_test_world_cmd + + # TODO: We want to include crashlogs, but they are not yet + # collected. cdb.exe is installed on the runner, so we can configure it + # appropriately. + - *upload_logs_step + + + # Job: CompilerWarnings + # + # Test that code can be built with both gcc and clang without warnings, + # with various combinations of cassert/dtrace flags. Trace probes have + # a history of getting accidentally broken; the matrix is there to + # catch that. + # + # The autoconf cache files (gcc.cache / clang.cache) are intentionally + # reused across the matrix entries that share a compiler, so we don't + # pay for full feature detection on every entry. + compiler-warnings: + name: CompilerWarnings + needs: [setup, sanity-check] + if: | + !cancelled() && + needs.setup.outputs.compilerwarnings == 'true' && + needs.sanity-check.result != 'failure' + runs-on: *linux_runs_on + timeout-minutes: 60 + container: + image: ${{ needs.setup.outputs.container_linux_ci_docs }} + env: + # Use larger ccache cache as this job compiles with multiple + # compilers / flag combinations. + CCACHE_MAXSIZE: "1G" + DEFAULT_BUILD: world-bin + + steps: + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + + - name: Setup workspace + run: | + echo "COPT=-Werror" > src/Makefile.custom + + # gcc, cassert off, dtrace on + - name: gcc warnings + (dtrace) + if: ${{ !cancelled() }} + env: + CONF: ${{env.LINUX_CONFIGURE_FEATURES}} --cache gcc.cache --enable-dtrace + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + run: &compiler_warnings_cmd | + echo "::group::configure" + ./configure \ + ${{env.CONF}} \ + CLANG="ccache clang" + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} ${{env.DEFAULT_BUILD}} + + # gcc, cassert on, dtrace off + - name: gcc warnings + (cassert) + if: ${{ !cancelled() }} + env: + CONF: ${{env.LINUX_CONFIGURE_FEATURES}} --cache gcc.cache --enable-cassert + CC: ccache gcc + CXX: ccache g++ + run: *compiler_warnings_cmd + + # clang, cassert off, dtrace off + - name: clang warnings + if: ${{ !cancelled() }} + env: + CONF: ${{env.LINUX_CONFIGURE_FEATURES}} --cache clang.cache + CC: ccache clang + CXX: ccache clang++ + run: *compiler_warnings_cmd + + # clang, cassert on, dtrace on + - name: clang warnings + (cassert + dtrace) + if: ${{ !cancelled() }} + env: + CONF: ${{env.LINUX_CONFIGURE_FEATURES}} --cache clang.cache --enable-cassert --enable-dtrace + CC: ccache clang + CXX: ccache clang++ + run: *compiler_warnings_cmd + + - name: mingw warnings (cross compilation) + if: ${{ !cancelled() }} + env: + CONF: --host=x86_64-w64-mingw32ucrt --enable-cassert --without-icu + CC: ccache x86_64-w64-mingw32ucrt-gcc + CXX: ccache x86_64-w64-mingw32ucrt-g++ + run: *compiler_warnings_cmd + + ### + # Verify docs can be built + ### + # XXX: Only do this if there have been changes in doc/ since last build + - name: Build documentation + if: ${{ !cancelled() }} + env: + CONF: --cache gcc.cache + CC: ccache gcc + CXX: ccache g++ + DEFAULT_BUILD: -C doc + run: *compiler_warnings_cmd + + ### + # Verify headerscheck / cpluspluscheck succeed + # + # - Run both in same script to increase parallelism, use -k to get + # result of both + # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose + ### + - name: headerscheck + cpluspluscheck + if: ${{ !cancelled() }} + run: | + echo "::group::configure" + ./configure \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ + --cache gcc.cache \ + --quiet \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step + - *upload_logs_step diff --git a/src/tools/ci/README b/src/tools/ci/README index d183648a8d0..d5c95f6a6ed 100644 --- a/src/tools/ci/README +++ b/src/tools/ci/README @@ -17,42 +17,52 @@ Postgres has two forms of CI: Configuring CI on personal repositories ======================================= -Currently postgres contains CI support utilizing cirrus-ci. cirrus-ci -currently is only available for github. +Currently postgres contains CI support utilizing GitHub Actions. -Enabling cirrus-ci in a github repository +Configuring CI use of a GitHub repository ========================================= -To enable cirrus-ci on a repository, go to -https://github.com/marketplace/cirrus-ci and select "Public -Repositories". Then "Install it for free" and "Complete order". The next page -allows to configure which repositories cirrus-ci has access to. Choose the -relevant repository and "Install". +The GitHub Actions based CI workflow may or may not be active by default, +depending on when the repository was forked. -See also https://cirrus-ci.org/guide/quick-start/ +To disable the CI workflow on a repository, navigate to +https://github.com/<username>/<reponame>/actions/workflows/pg-ci.yml +and click on the '...' on the top right and choose 'Disable workflow'. -Once enabled on a repository, future commits and pull-requests in that -repository will automatically trigger CI builds. These are visible from the -commit history / PRs, and can also be viewed in the cirrus-ci UI at -https://cirrus-ci.com/github/<username>/<reponame>/ +To enable the workflow, go to the same page and click on "Enable workflow" at +the top. -Hint: all build log files are uploaded to cirrus-ci and can be downloaded -from the "Artifacts" section from the cirrus-ci UI after clicking into a -specific task on a build's summary page. +However, to avoid issues with the thousands of forks of the postgres/postgres +repository starting to run CI the next time the forks re-synchronize with the +postgres/postgres, each repository needs to explicitly opt-in to actually run +the full CI tests. +To opt into CI, go to +https://github.com/<username>/<reponame>//settings/variables/actions and +create a new repository variable named PG_CI_ENABLED, with the value 1. -Images used for CI -================== -To keep CI times tolerable, most platforms use pre-generated images. Some -platforms use containers, others use full VMs. Images for both are generated -separately from CI runs, otherwise each git repository that is being tested -would need to build its own set of containers, which would be wasteful (both -in space and time. +Viewing CI results in a GitHub repository +========================================= + +CI runs are visible at https://github.com/<username>/<reponame>/actions + +The high-level status of workflow runs on public repositories are visible +without being logged into GitHub, however details including logs require being +logged in. + + +Containers / Images used for CI +=============================== + +To keep CI times tolerable, several platforms use pre-generated containers / +images. The containers and images are generated separately from CI runs, +otherwise each git repository that is being tested would need to build its own +set of containers, which would be wasteful (both in space and time). -These images are built, on a daily basis, from the specifications in -github.com/anarazel/pg-vm-images/ +These containers / images are built, on a daily basis, from the specifications +in github.com/anarazel/pg-vm-images/ Controlling CI via commit messages @@ -61,35 +71,7 @@ Controlling CI via commit messages The behavior of CI can be controlled by special content in commit messages. Currently the following controls are available: -- ci-os-only: {(freebsd|linux|macos|mingw|netbsd|openbsd|windows)} +- ci-os-only: {(compilerwarnings|linux|macos|mingw|sanitycheck|windows)} Only runs CI on operating systems specified. This can be useful when addressing portability issues affecting only a subset of platforms. - - -Using custom compute resources for CI -===================================== - -When running a lot of tests in a repository, cirrus-ci's free credits do not -suffice. In those cases a repository can be configured to use other -infrastructure for running tests. To do so, the REPO_CI_CONFIG_GIT_URL -variable can be configured for the repository in the cirrus-ci web interface, -at https://cirrus-ci.com/github/<user or organization>. The file referenced -(see https://cirrus-ci.org/guide/programming-tasks/#fs) by the variable can -overwrite the default execution method for different operating systems, -defined in .cirrus.yml, by redefining the relevant yaml anchors. - -Custom compute resources can be provided using -- https://cirrus-ci.org/guide/supported-computing-services/ -- https://cirrus-ci.org/guide/persistent-workers/ - - -Enabling manual tasks by default -================================ - -Some tasks are not triggered automatically by default, to avoid using up CI -credits too quickly. This can be changed on the repository level, e.g. when -custom compute resources are configured. - -The following repository level environment variables are recognized: -- REPO_CI_AUTOMATIC_TRIGGER_TASKS - space-separated list of (mingw|netbsd|openbsd) diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index 63e97b37c78..e49f4f703a0 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -6,7 +6,8 @@ # when packages are installed or removed. Any package this script is # not instructed to install, will be removed again. # -# This currently expects to be run in a macos cirrus-ci environment. +# This currently expects to be run in a GitHub Actions or cirrus-ci +# macOS environment. set -e # set -x @@ -20,13 +21,26 @@ echo "macOS major version: $macos_major_version" # macOS release. macports_release_list_url="https://api.github.com/repos/macports/macports-base/releases"; macports_version_pattern="2\.10\.1" -macports_url="$( curl -s $macports_release_list_url | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$macports_version_pattern-$macos_major_version-[A-Za-z]*\.pkg\""; | sed 's/.*: "//;s/".*//' | head -1 )" +# Authenticate the GitHub API request when a token is available (e.g. on +# GitHub Actions). Unauthenticated requests share a 60/h/IP rate limit +# with every other job on the runner's IP and frequently return an error +# JSON, leaving $macports_url empty and breaking the subsequent curl. +auth_header="" +if [ -n "$GITHUB_TOKEN" ]; then + auth_header="Authorization: Bearer $GITHUB_TOKEN" +fi +macports_url="$( curl -fsSL ${auth_header:+-H "$auth_header"} "$macports_release_list_url" | grep "\"https://github.com/macports/macports-base/releases/download/v$macports_version_pattern/MacPorts-$macports_version_pattern-$macos_major_version-[A-Za-z]*\.pkg\""; | sed 's/.*: "//;s/".*//' | head -1 )" echo "MacPorts package URL: $macports_url" +if [ -z "$macports_url" ]; then + echo "error: could not determine MacPorts package URL for macOS $macos_major_version (version pattern: $macports_version_pattern)" 1>&2 + exit 1 +fi + cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ]; then - echo "expect to be called within cirrus-ci" 1>2 +if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within cirrus-ci or GitHub Actions" 1>&2 exit 1 fi -- 2.54.0.380.gc69baaf57b
>From 1f3418614609e1ad8678dac8333599d8b51e2eb9 Mon Sep 17 00:00:00 2001 From: Andres Freund <[email protected]> Date: Thu, 4 Jun 2026 09:34:50 -0400 Subject: [PATCH v12a 2/3] ci: Remove support for cirrus-ci based CI As mentioned in the earlier commit, cirrus-ci has shut down. Therefore remove all files related to running CI via cirrus. Also update comments / code that were referencing cirrus-ci. Discussion: https://postgr.es/m/3ydjipcr7kbss57nvi67noplncqhesl5eyb6wgol4ccjxynspv%40yatlykpribmm --- src/bin/pg_combinebackup/t/010_hardlink.pl | 12 +- .cirrus.yml | 91 -- src/test/perl/PostgreSQL/Test/Cluster.pm | 5 +- .cirrus.star | 143 --- .cirrus.tasks.yml | 1022 -------------------- src/tools/ci/ci_macports_packages.sh | 7 +- src/tools/ci/gcp_ram_disk.sh | 27 - 7 files changed, 11 insertions(+), 1296 deletions(-) delete mode 100644 .cirrus.yml delete mode 100644 .cirrus.star delete mode 100644 .cirrus.tasks.yml delete mode 100755 src/tools/ci/gcp_ram_disk.sh diff --git a/src/bin/pg_combinebackup/t/010_hardlink.pl b/src/bin/pg_combinebackup/t/010_hardlink.pl index b6e8a9128af..5fbbe6a923b 100644 --- a/src/bin/pg_combinebackup/t/010_hardlink.pl +++ b/src/bin/pg_combinebackup/t/010_hardlink.pl @@ -18,13 +18,13 @@ $primary->append_conf('postgresql.conf', 'autovacuum = off'); $primary->start; # Create a couple of tables (~264KB each). -# Note: Cirrus CI runs some tests with a very small segment size, so, in that +# Note: CI runs some tests with a very small segment size, so, in that # environment, a single table of 264KB would have both a segment with a link -# count of 1 and also one with a link count of 2. But in a normal installation, -# segment size is 1GB. Therefore, we use 2 different tables here: for test_1, -# all segments (or the only one) will have two hard links; for test_2, the -# last segment (or the only one) will have 1 hard link, and any others will -# have 2. +# count of 1 and also one with a link count of 2. But in a normal +# installation, segment size is 1GB. Therefore, we use 2 different tables +# here: for test_1, all segments (or the only one) will have two hard links; +# for test_2, the last segment (or the only one) will have 1 hard link, and +# any others will have 2. my $query = <<'EOM'; CREATE TABLE test_%s AS SELECT x.id::bigint, diff --git a/.cirrus.yml b/.cirrus.yml deleted file mode 100644 index 3f75852e84e..00000000000 --- a/.cirrus.yml +++ /dev/null @@ -1,91 +0,0 @@ -# CI configuration file for CI utilizing cirrus-ci.org -# -# For instructions on how to enable the CI integration in a repository and -# further details, see src/tools/ci/README -# -# -# The actual CI tasks are defined in .cirrus.tasks.yml. To make the compute -# resources for CI configurable on a repository level, the "final" CI -# configuration is the combination of: -# -# 1) the contents of this file -# -# 2) computed environment variables -# -# Used to enable/disable tasks based on the execution environment. See -# .cirrus.star: compute_environment_vars() -# -# 3) if defined, the contents of the file referenced by the, repository -# level, REPO_CI_CONFIG_GIT_URL variable (see -# https://cirrus-ci.org/guide/programming-tasks/#fs for the accepted -# format) -# -# This allows running tasks in a different execution environment than the -# default, e.g. to have sufficient resources for cfbot. -# -# 4) .cirrus.tasks.yml -# -# This composition is done by .cirrus.star - - -env: - # Source of images / containers - GCP_PROJECT: pg-ci-images - IMAGE_PROJECT: $GCP_PROJECT - CONTAINER_REPO: us-docker.pkg.dev/${GCP_PROJECT}/ci - DISK_SIZE: 25 - - -# Define how to run various types of tasks. - -# VMs provided by cirrus-ci. Each user has a limited number of "free" credits -# for testing. -cirrus_community_vm_template: &cirrus_community_vm_template - compute_engine_instance: - image_project: $IMAGE_PROJECT - image: family/$IMAGE_FAMILY - platform: $PLATFORM - cpu: $CPUS - disk: $DISK_SIZE - - -default_linux_task_template: &linux_task_template - env: - PLATFORM: linux - <<: *cirrus_community_vm_template - - -default_freebsd_task_template: &freebsd_task_template - env: - PLATFORM: freebsd - <<: *cirrus_community_vm_template - -default_netbsd_task_template: &netbsd_task_template - env: - PLATFORM: netbsd - <<: *cirrus_community_vm_template - -default_openbsd_task_template: &openbsd_task_template - env: - PLATFORM: openbsd - <<: *cirrus_community_vm_template - - -default_windows_task_template: &windows_task_template - env: - PLATFORM: windows - <<: *cirrus_community_vm_template - - -# macos workers provided by cirrus-ci -default_macos_task_template: &macos_task_template - env: - PLATFORM: macos - macos_instance: - image: $IMAGE - - -# Contents of REPO_CI_CONFIG_GIT_URL, if defined, will be inserted here, -# followed by the contents .cirrus.tasks.yml. This allows -# REPO_CI_CONFIG_GIT_URL to override how the task types above will be -# executed, e.g. using a custom compute account or permanent workers. diff --git a/src/test/perl/PostgreSQL/Test/Cluster.pm b/src/test/perl/PostgreSQL/Test/Cluster.pm index 4fcb1f6be56..529f49efee1 100644 --- a/src/test/perl/PostgreSQL/Test/Cluster.pm +++ b/src/test/perl/PostgreSQL/Test/Cluster.pm @@ -363,9 +363,8 @@ This tries to connect to the server, to test whether it works or not,, so the server is up and running. Otherwise this can return 0 even if there's nothing wrong with raw_connect() itself. -Notably, raw_connect() does not work on Unix domain sockets on -Strawberry perl 5.26.3.1 on Windows, which we use in Cirrus CI images -as of this writing. It dies with "not implemented on this +Notably, raw_connect() does not work on Unix domain sockets on at least +Strawberry perl 5.26.3.1 on Windows. It dies with "not implemented on this architecture". =cut diff --git a/.cirrus.star b/.cirrus.star deleted file mode 100644 index e9bb672b959..00000000000 --- a/.cirrus.star +++ /dev/null @@ -1,143 +0,0 @@ -"""Additional CI configuration, using the starlark language. See -https://cirrus-ci.org/guide/programming-tasks/#introduction-into-starlark - -See also the starlark specification at -https://github.com/bazelbuild/starlark/blob/master/spec.md - -See also .cirrus.yml and src/tools/ci/README -""" - -load("cirrus", "env", "fs", "re", "yaml") - - -def main(): - """The main function is executed by cirrus-ci after loading .cirrus.yml and can - extend the CI definition further. - - As documented in .cirrus.yml, the final CI configuration is composed of - - 1) the contents of .cirrus.yml - - 2) computed environment variables - - 3) if defined, the contents of the file referenced by the, repository - level, REPO_CI_CONFIG_GIT_URL variable (see - https://cirrus-ci.org/guide/programming-tasks/#fs for the accepted - format) - - 4) .cirrus.tasks.yml - """ - - output = "" - - # 1) is evaluated implicitly - - - # Add 2) - additional_env = compute_environment_vars() - env_fmt = """ -### -# Computed environment variables start here -### -{0} -### -# Computed environment variables end here -### -""" - output += env_fmt.format(yaml.dumps({'env': additional_env})) - - - # Add 3) - repo_config_url = env.get("REPO_CI_CONFIG_GIT_URL") - if repo_config_url != None: - print("loading additional configuration from \"{}\"".format(repo_config_url)) - output += config_from(repo_config_url) - else: - output += "\n# REPO_CI_CONFIG_URL was not set\n" - - - # Add 4) - output += config_from(".cirrus.tasks.yml") - - - return output - - -def compute_environment_vars(): - cenv = {} - - ### - # Some tasks are manually triggered by default because they might use too - # many resources for users of free Cirrus credits, but they can be - # triggered automatically by naming them in an environment variable e.g. - # REPO_CI_AUTOMATIC_TRIGGER_TASKS="task_name other_task" under "Repository - # Settings" on Cirrus CI's website. - - default_manual_trigger_tasks = ['mingw', 'netbsd', 'openbsd'] - - repo_ci_automatic_trigger_tasks = env.get('REPO_CI_AUTOMATIC_TRIGGER_TASKS', '') - for task in default_manual_trigger_tasks: - name = 'CI_TRIGGER_TYPE_' + task.upper() - if repo_ci_automatic_trigger_tasks.find(task) != -1: - value = 'automatic' - else: - value = 'manual' - cenv[name] = value - ### - - ### - # Parse "ci-os-only:" tag in commit message and set - # CI_{$OS}_ENABLED variable for each OS - - # We want to disable SanityCheck if testing just a specific OS. This - # shortens push-wait-for-ci cycle time a bit when debugging operating - # system specific failures. Just treating it as an OS in that case - # suffices. - - operating_systems = [ - 'compilerwarnings', - 'freebsd', - 'linux', - 'macos', - 'mingw', - 'netbsd', - 'openbsd', - 'sanitycheck', - 'windows', - ] - commit_message = env.get('CIRRUS_CHANGE_MESSAGE') - match_re = r"(^|.*\n)ci-os-only: ([^\n]+)($|\n.*)" - - # re.match() returns an array with a tuple of (matched-string, match_1, ...) - m = re.match(match_re, commit_message) - if m and len(m) > 0: - os_only = m[0][2] - os_only_list = re.split(r'[, ]+', os_only) - else: - os_only_list = operating_systems - - for os in operating_systems: - os_enabled = os in os_only_list - cenv['CI_{0}_ENABLED'.format(os.upper())] = os_enabled - ### - - return cenv - - -def config_from(config_src): - """return contents of config file `config_src`, surrounded by markers - indicating start / end of the included file - """ - - config_contents = fs.read(config_src) - config_fmt = """ - -### -# contents of config file `{0}` start here -### -{1} -### -# contents of config file `{0}` end here -### -""" - return config_fmt.format(config_src, config_contents) diff --git a/.cirrus.tasks.yml b/.cirrus.tasks.yml deleted file mode 100644 index 8683d1ae9c7..00000000000 --- a/.cirrus.tasks.yml +++ /dev/null @@ -1,1022 +0,0 @@ -# CI configuration file for CI utilizing cirrus-ci.org -# -# For instructions on how to enable the CI integration in a repository and -# further details, see src/tools/ci/README -# -# -# NB: Different tasks intentionally test with different, non-default, -# configurations, to increase the chance of catching problems. Each task with -# non-obvious non-default documents their oddity at the top of the task, -# prefixed by "SPECIAL:". - - -env: - # The lower depth accelerates git clone. Use a bit of depth so that - # concurrent tasks and retrying older jobs have a chance of working. - CIRRUS_CLONE_DEPTH: 500 - # Useful to be able to analyse what in a script takes long - CIRRUS_LOG_TIMESTAMP: true - - CCACHE_MAXSIZE: "250M" - - # target to test, for all but windows - CHECK: check-world PROVE_FLAGS=$PROVE_FLAGS - CHECKFLAGS: -Otarget - PROVE_FLAGS: --timer - # Build test dependencies as part of the build step, to see compiler - # errors/warnings in one place. - MBUILD_TARGET: all testprep - MTEST_ARGS: --print-errorlogs --no-rebuild -C build - PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests - TEMP_CONFIG: ${CIRRUS_WORKING_DIR}/src/tools/ci/pg_ci_base.conf - PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth - - # Postgres config args for the meson builds, shared between all meson tasks - # except the 'SanityCheck' task - MESON_COMMON_PG_CONFIG_ARGS: -Dcassert=true -Dinjection_points=true - - # Meson feature flags shared by all meson tasks, except: - # SanityCheck: uses almost no dependencies. - # Windows - VS: has fewer dependencies than listed here, so defines its own. - # Linux: uses the 'auto' feature option to test meson feature autodetection. - MESON_COMMON_FEATURES: >- - -Dauto_features=disabled - -Dldap=enabled - -Dssl=openssl - -Dtap_tests=enabled - -Dplperl=enabled - -Dplpython=enabled - -Ddocs=enabled - -Dicu=enabled - -Dlibxml=enabled - -Dlibxslt=enabled - -Dlz4=enabled - -Dpltcl=enabled - -Dreadline=enabled - -Dzlib=enabled - -Dzstd=enabled - - -# What files to preserve in case tests fail -on_failure_ac: &on_failure_ac - log_artifacts: - paths: - - "**/*.log" - - "**/*.diffs" - - "**/regress_log_*" - type: text/plain - -on_failure_meson: &on_failure_meson - testrun_artifacts: - paths: - - "build*/testrun/**/*.log" - - "build*/testrun/**/*.diffs" - - "build*/testrun/**/regress_log_*" - type: text/plain - - # In theory it'd be nice to upload the junit files meson generates, so that - # cirrus will nicely annotate the commit. Unfortunately the files don't - # contain identifiable file + line numbers right now, so the annotations - # don't end up useful. We could probably improve on that with a some custom - # conversion script, but ... - meson_log_artifacts: - path: "build*/meson-logs/*.txt" - type: text/plain - - -# To avoid unnecessarily spinning up a lot of VMs / containers for entirely -# broken commits, have a minimal task that all others depend on. -# -# SPECIAL: -# - Builds with --auto-features=disabled and thus almost no enabled -# dependencies -task: - name: SanityCheck - - # If a specific OS is requested, don't run the sanity check. This shortens - # push-wait-for-ci cycle time a bit when debugging operating system specific - # failures. Uses skip instead of only_if, as cirrus otherwise warns about - # only_if conditions not matching. - skip: $CI_SANITYCHECK_ENABLED == false - - env: - CPUS: 4 - BUILD_JOBS: 8 - TEST_JOBS: 8 - IMAGE_FAMILY: pg-ci-trixie - CCACHE_DIR: ${CIRRUS_WORKING_DIR}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - - # While containers would start up a bit quicker, building is a bit - # slower. This way we don't have to maintain a container image. - <<: *linux_task_template - - ccache_cache: - folder: $CCACHE_DIR - - create_user_script: | - useradd -m postgres - chown -R postgres:postgres . - mkdir -p ${CCACHE_DIR} - chown -R postgres:postgres ${CCACHE_DIR} - echo '* - memlock 134217728' > /etc/security/limits.d/postgres.conf - su postgres -c "ulimit -l -H && ulimit -l -S" - # Can't change container's kernel.core_pattern. Postgres user can't write - # to / normally. Change that. - chown root:postgres / - chmod g+rwx / - - configure_script: | - su postgres <<-EOF - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - build_script: | - su postgres <<-EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - upload_caches: ccache - - # Run a minimal set of tests. The main regression tests take too long for - # this purpose. For now this is a random quick pg_regress style test, and a - # tap test that exercises both a frontend binary and the backend. - test_minimal_script: | - su postgres <<-EOF - set -e - ulimit -c unlimited - meson test $MTEST_ARGS --suite setup - meson test $MTEST_ARGS --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - on_failure: - <<: *on_failure_meson - cores_script: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - -# SPECIAL: -# - Uses postgres specific CPPFLAGS that increase test coverage -# - Specifies configuration options that test reading/writing/copying of node trees -# - Specifies debug_parallel_query=regress, to catch related issues during CI -# - Also runs tests against a running postgres instance, see test_running_script -task: - name: FreeBSD - Meson - - env: - CPUS: 4 - BUILD_JOBS: 4 - TEST_JOBS: 8 - IMAGE_FAMILY: pg-ci-freebsd - DISK_SIZE: 50 - - CCACHE_DIR: /tmp/ccache_dir - CPPFLAGS: -DRELCACHE_FORCE_RELEASE -DENFORCE_REGRESSION_TEST_NAME_RESTRICTIONS - CFLAGS: -Og -ggdb - - # Several buildfarm animals enable these options. Without testing them - # during CI, it would be easy to cause breakage on the buildfarm with CI - # passing. - PG_TEST_INITDB_EXTRA_OPTS: >- - -c debug_copy_parse_plan_trees=on - -c debug_write_read_parse_plan_trees=on - -c debug_raw_expression_coverage_test=on - -c debug_parallel_query=regress - PG_TEST_PG_UPGRADE_MODE: --link - - MESON_FEATURES: >- - -Ddtrace=enabled - -Dgssapi=enabled - -Dlibcurl=enabled - -Dnls=enabled - -Dpam=enabled - -Dtcl_version=tcl86 - -Duuid=bsd - - <<: *freebsd_task_template - - depends_on: SanityCheck - only_if: $CI_FREEBSD_ENABLED - - sysinfo_script: | - id - uname -a - ulimit -a -H && ulimit -a -S - export - - ccache_cache: - folder: $CCACHE_DIR - setup_ram_disk_script: src/tools/ci/gcp_ram_disk.sh - create_user_script: | - pw useradd postgres - chown -R postgres:postgres . - mkdir -p ${CCACHE_DIR} - chown -R postgres:postgres ${CCACHE_DIR} - setup_core_files_script: | - mkdir -m 770 /tmp/cores - chown root:postgres /tmp/cores - sysctl kern.corefile='/tmp/cores/%N.%P.core' - setup_additional_packages_script: | - #pkg install -y ... - - # NB: Intentionally build without -Dllvm. The freebsd image size is already - # large enough to make VM startup slow, and even without llvm freebsd - # already takes longer than other platforms except for windows. - configure_script: | - su postgres <<-EOF - set -e - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - --buildtype=debug \ - -Dextra_lib_dirs=/usr/local/lib -Dextra_include_dirs=/usr/local/include/ \ - ${MESON_COMMON_FEATURES} ${MESON_FEATURES} \ - build - EOF - build_script: su postgres -c 'ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET}' - upload_caches: ccache - - test_world_script: | - su postgres <<-EOF - set -e - ulimit -c unlimited - meson test $MTEST_ARGS --num-processes ${TEST_JOBS} - EOF - - # test runningcheck, freebsd chosen because it's currently fast enough - test_running_script: | - su postgres <<-EOF - set -e - ulimit -c unlimited - meson test $MTEST_ARGS --quiet --suite setup - export LD_LIBRARY_PATH="$(pwd)/build/tmp_install/usr/local/pgsql/lib/:$LD_LIBRARY_PATH" - mkdir -p build/testrun - build/tmp_install/usr/local/pgsql/bin/initdb -N build/runningcheck --no-instructions -A trust - echo "include '$(pwd)/src/tools/ci/pg_ci_base.conf'" >> build/runningcheck/postgresql.conf - build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start - meson test $MTEST_ARGS --num-processes ${TEST_JOBS} --setup running - build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop - EOF - - on_failure: - # if the server continues running, it often causes cirrus-ci to fail - # during upload, as it doesn't expect artifacts to change size - stop_running_script: | - su postgres <<-EOF - set -e - build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop || true - EOF - <<: *on_failure_meson - cores_script: src/tools/ci/cores_backtrace.sh freebsd /tmp/cores - - -task: - depends_on: SanityCheck - - env: - # Below are experimentally derived to be a decent choice. - CPUS: 4 - BUILD_JOBS: 8 - TEST_JOBS: 8 - - # Default working directory is /tmp, but its total size (1.2 GB) is not - # enough, so different working and cache directory are set. - CIRRUS_WORKING_DIR: /home/postgres/postgres - CCACHE_DIR: /home/postgres/cache - - PATH: /usr/sbin:$PATH - CORE_DUMP_DIR: /var/crash - - matrix: - - name: NetBSD - Meson - # See REPO_CI_AUTOMATIC_TRIGGER_TASKS in .cirrus.star - trigger_type: $CI_TRIGGER_TYPE_NETBSD - only_if: $CI_NETBSD_ENABLED - env: - OS_NAME: netbsd - IMAGE_FAMILY: pg-ci-netbsd-postgres - PKGCONFIG_PATH: '/usr/lib/pkgconfig:/usr/pkg/lib/pkgconfig' - # initdb fails with: 'invalid locale settings' error on NetBSD. - # Force 'LANG' and 'LC_*' variables to be 'C'. - # See https://postgr.es/m/2490325.1734471752%40sss.pgh.pa.us - LANG: "C" - LC_ALL: "C" - # -Duuid is not set for the NetBSD, see the comment below, above - # configure_script, for more information. - MESON_FEATURES: >- - -Dgssapi=enabled - -Dlibcurl=enabled - -Dnls=enabled - -Dpam=enabled - - setup_additional_packages_script: | - #pkgin -y install ... - <<: *netbsd_task_template - - - name: OpenBSD - Meson - # See REPO_CI_AUTOMATIC_TRIGGER_TASKS in .cirrus.star - trigger_type: $CI_TRIGGER_TYPE_OPENBSD - only_if: $CI_OPENBSD_ENABLED - env: - OS_NAME: openbsd - IMAGE_FAMILY: pg-ci-openbsd-postgres - PKGCONFIG_PATH: '/usr/lib/pkgconfig:/usr/local/lib/pkgconfig' - CORE_DUMP_EXECUTABLE_DIR: $CIRRUS_WORKING_DIR/build/tmp_install/usr/local/pgsql/bin - - MESON_FEATURES: >- - -Dbsd_auth=enabled - -Dlibcurl=enabled - -Dtcl_version=tcl86 - -Duuid=e2fs - - setup_additional_packages_script: | - #pkg_add -I ... - # Always core dump to ${CORE_DUMP_DIR} - set_core_dump_script: sysctl -w kern.nosuidcoredump=2 - <<: *openbsd_task_template - - sysinfo_script: | - locale - id - uname -a - ulimit -a -H && ulimit -a -S - env - - ccache_cache: - folder: $CCACHE_DIR - setup_ram_disk_script: src/tools/ci/gcp_ram_disk.sh - create_user_script: | - useradd postgres - chown -R postgres:users /home/postgres - mkdir -p ${CCACHE_DIR} - chown -R postgres:users ${CCACHE_DIR} - setup_core_files_script: | - mkdir -p ${CORE_DUMP_DIR} - chmod -R 770 ${CORE_DUMP_DIR} - chown -R postgres:users ${CORE_DUMP_DIR} - - # -Duuid=bsd is not set since 'bsd' uuid option - # is not working on NetBSD & OpenBSD. See - # https://www.postgresql.org/message-id/[email protected] - # And other uuid options are not available on NetBSD. - configure_script: | - su postgres <<-EOF - set -e - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - --buildtype=debugoptimized \ - --pkg-config-path ${PKGCONFIG_PATH} \ - ${MESON_COMMON_FEATURES} ${MESON_FEATURES} \ - build - EOF - - build_script: su postgres -c 'ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET}' - upload_caches: ccache - - test_world_script: | - su postgres <<-EOF - set -e - ulimit -c unlimited - meson test $MTEST_ARGS --num-processes ${TEST_JOBS} - EOF - - on_failure: - <<: *on_failure_meson - cores_script: | - # Although we try to configure the OS to core dump inside - # ${CORE_DUMP_DIR}, they may not obey this. So, move core files to the - # ${CORE_DUMP_DIR} directory. - find build/ -type f -name '*.core' -exec mv '{}' ${CORE_DUMP_DIR} \; - src/tools/ci/cores_backtrace.sh ${OS_NAME} ${CORE_DUMP_DIR} ${CORE_DUMP_EXECUTABLE_DIR} - - -# configure feature flags, shared between the task running the linux tests and -# the CompilerWarnings task -LINUX_CONFIGURE_FEATURES: &LINUX_CONFIGURE_FEATURES >- - --with-gssapi - --with-icu - --with-ldap - --with-libcurl - --with-libxml - --with-libxslt - --with-llvm - --with-lz4 - --with-pam - --with-perl - --with-python - --with-selinux - --with-ssl=openssl - --with-systemd - --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ - --with-uuid=ossp - --with-zstd - - -# Check SPECIAL in the matrix: below -task: - env: - CPUS: 4 - BUILD_JOBS: 4 - TEST_JOBS: 8 # experimentally derived to be a decent choice - IMAGE_FAMILY: pg-ci-trixie - - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - # Enable a reasonable set of sanitizers. Use the linux task for that, as - # it's one of the fastest tasks (without sanitizers). Also several of the - # sanitizers work best on linux. - # - # The overhead of alignment sanitizer is low, undefined behaviour has - # moderate overhead. Test alignment sanitizer in the meson task, as it - # does both 32 and 64 bit builds and is thus more likely to expose - # alignment bugs. - # - # Address sanitizer in contrast is somewhat expensive. Enable it in the - # autoconf task, as the meson task tests both 32 and 64bit. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 - - # SANITIZER_FLAGS is set in the tasks below - CFLAGS: -Og -ggdb -fno-sanitize-recover=all $SANITIZER_FLAGS - CXXFLAGS: $CFLAGS - LDFLAGS: $SANITIZER_FLAGS - CC: ccache gcc - CXX: ccache g++ - - LINUX_CONFIGURE_FEATURES: *LINUX_CONFIGURE_FEATURES - LINUX_MESON_FEATURES: >- - -Duuid=e2fs - - <<: *linux_task_template - - depends_on: SanityCheck - only_if: $CI_LINUX_ENABLED - - ccache_cache: - folder: ${CCACHE_DIR} - - sysinfo_script: | - id - uname -a - cat /proc/cmdline - ulimit -a -H && ulimit -a -S - export - create_user_script: | - useradd -m postgres - chown -R postgres:postgres . - mkdir -p ${CCACHE_DIR} - chown -R postgres:postgres ${CCACHE_DIR} - echo '* - memlock 134217728' > /etc/security/limits.d/postgres.conf - su postgres -c "ulimit -l -H && ulimit -l -S" - setup_core_files_script: | - mkdir -m 770 /tmp/cores - chown root:postgres /tmp/cores - sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' - - setup_hosts_file_script: | - cat >> /etc/hosts <<-EOF - 127.0.0.1 pg-loadbalancetest - 127.0.0.2 pg-loadbalancetest - 127.0.0.3 pg-loadbalancetest - EOF - - setup_additional_packages_script: | - #apt-get update - #DEBIAN_FRONTEND=noninteractive apt-get -y install ... - - matrix: - # SPECIAL: - # - Uses address sanitizer, sanitizer failures are typically printed in - # the server log - # - Configures postgres with a small segment size - - name: Linux - Debian Trixie - Autoconf - - env: - SANITIZER_FLAGS: -fsanitize=address - PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range - - # Normally, the "relation segment" code basically has no coverage in our - # tests, because we (quite reasonably) don't generate tables large - # enough in tests. We've had plenty bugs that we didn't notice due the - # code not being exercised much. Thus specify a very small segment size - # here. Use a non-power-of-two segment size, given we currently allow - # that. - configure_script: | - su postgres <<-EOF - set -e - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - \ - ${LINUX_CONFIGURE_FEATURES} \ - \ - CLANG="ccache clang" - EOF - build_script: su postgres -c "make -s -j${BUILD_JOBS} world-bin" - upload_caches: ccache - - test_world_script: | - su postgres <<-EOF - set -e - ulimit -c unlimited # default is 0 - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - EOF - - on_failure: - <<: *on_failure_ac - - # SPECIAL: - # - Uses undefined behaviour and alignment sanitizers, sanitizer failures - # are typically printed in the server log - # - Test both 64bit and 32 bit builds - # - uses io_method=io_uring - # - Uses meson feature autodetection - - name: Linux - Debian Trixie - Meson - - env: - CCACHE_MAXSIZE: "400M" # tests two different builds - SANITIZER_FLAGS: -fsanitize=alignment,undefined - PG_TEST_INITDB_EXTRA_OPTS: >- - -c io_method=io_uring - - configure_script: | - su postgres <<-EOF - set -e - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - --buildtype=debug \ - ${LINUX_MESON_FEATURES} -Dllvm=enabled \ - build - EOF - - # Also build & test in a 32bit build - it's gotten rare to test that - # locally. - configure_32_script: | - su postgres <<-EOF - set -e - export CC='ccache gcc -m32' - export CXX='ccache g++ -m32' - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - ${LINUX_MESON_FEATURES} -Dlibnuma=disabled \ - build-32 - EOF - - build_script: | - su postgres <<-EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - EOF - - build_32_script: | - su postgres <<-EOF - set -e - ninja -C build-32 -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - EOF - - upload_caches: ccache - - test_world_script: | - su postgres <<-EOF - set -e - ulimit -c unlimited - meson test $MTEST_ARGS --num-processes ${TEST_JOBS} - EOF - # so that we don't upload 64bit logs if 32bit fails - rm -rf build/ - - # There's currently no coverage of icu with LANG=C in the buildfarm. We - # can easily provide some here by running one of the sets of tests that - # way. Newer versions of python insist on changing the LC_CTYPE away - # from C, prevent that with PYTHONCOERCECLOCALE. - test_world_32_script: | - su postgres <<-EOF - set -e - ulimit -c unlimited - PYTHONCOERCECLOCALE=0 LANG=C meson test $MTEST_ARGS -C build-32 --num-processes ${TEST_JOBS} - EOF - - on_failure: - <<: *on_failure_meson - - on_failure: - cores_script: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - -# NB: macOS is by far the most expensive OS to run CI for, therefore no -# expensive additional checks should be added. -# -# SPECIAL: -# - Enables --clone for pg_upgrade and pg_combinebackup -task: - name: macOS - Sequoia - Meson - - env: - CPUS: 4 # always get that much for cirrusci macOS instances - BUILD_JOBS: $CPUS - # Test performance regresses noticeably when using all cores. 8 seems to - # work OK. See - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - TEST_JOBS: 8 - IMAGE: ghcr.io/cirruslabs/macos-runner:sequoia - - CIRRUS_WORKING_DIR: ${HOME}/pgsql/ - CCACHE_DIR: ${HOME}/ccache - MACPORTS_CACHE: ${HOME}/macports-cache - - MESON_FEATURES: >- - -Dbonjour=enabled - -Ddtrace=enabled - -Dgssapi=enabled - -Dlibcurl=enabled - -Dnls=enabled - -Duuid=e2fs - - MACOS_PACKAGE_LIST: >- - ccache - icu - kerberos5 - lz4 - meson - openldap - openssl - p5.34-io-tty - p5.34-ipc-run - python312 - tcl - zstd - - CC: ccache cc - CXX: ccache c++ - CFLAGS: -Og -ggdb - CXXFLAGS: -Og -ggdb - - PG_TEST_PG_UPGRADE_MODE: --clone - PG_TEST_PG_COMBINEBACKUP_MODE: --clone - - <<: *macos_task_template - - depends_on: SanityCheck - only_if: $CI_MACOS_ENABLED - - sysinfo_script: | - id - uname -a - ulimit -a -H && ulimit -a -S - export - - setup_core_files_script: - - mkdir ${HOME}/cores - - sudo sysctl kern.corefile="${HOME}/cores/core.%P" - - # Use macports, even though homebrew is installed. The installation - # of the additional packages we need would take quite a while with - # homebrew, even if we cache the downloads. We can't cache all of - # homebrew, because it's already large. So we use macports. To cache - # the installation we create a .dmg file that we mount if it already - # exists. - # XXX: The reason for the direct p5.34* references is that we'd need - # the large macport tree around to figure out that p5-io-tty is - # actually p5.34-io-tty. Using the unversioned name works, but - # updates macports every time. - macports_cache: - folder: ${MACPORTS_CACHE} - fingerprint_script: | - # Reinstall packages if the OS major version, the list of the packages - # to install or the MacPorts install script changes. - sw_vers -productVersion | sed 's/\..*//' - echo $MACOS_PACKAGE_LIST - md5 src/tools/ci/ci_macports_packages.sh - reupload_on_changes: true - setup_additional_packages_script: | - sh src/tools/ci/ci_macports_packages.sh $MACOS_PACKAGE_LIST - # system python doesn't provide headers - sudo /opt/local/bin/port select python3 python312 - # Make macports install visible for subsequent steps - echo PATH=/opt/local/sbin/:/opt/local/bin/:$PATH >> $CIRRUS_ENV - upload_caches: macports - - ccache_cache: - folder: $CCACHE_DIR - configure_script: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - --buildtype=debug \ - -Dextra_include_dirs=/opt/local/include \ - -Dextra_lib_dirs=/opt/local/lib \ - ${MESON_COMMON_FEATURES} ${MESON_FEATURES} \ - build - - build_script: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - upload_caches: ccache - - test_world_script: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test $MTEST_ARGS --num-processes ${TEST_JOBS} - - on_failure: - <<: *on_failure_meson - cores_script: src/tools/ci/cores_backtrace.sh macos "${HOME}/cores" - - -WINDOWS_ENVIRONMENT_BASE: &WINDOWS_ENVIRONMENT_BASE - env: - # Half the allowed per-user CPU cores - CPUS: 4 - - # The default cirrus working dir is in a directory msbuild complains about - CIRRUS_WORKING_DIR: "c:/cirrus" - # git's tar doesn't deal with drive letters, see - # https://postgr.es/m/b6782dc3-a7b0-ed56-175f-f8f54cb08d67%40dunslane.net - TAR: "c:/windows/system32/tar.exe" - # Avoids port conflicts between concurrent tap test runs - PG_TEST_USE_UNIX_SOCKETS: 1 - PG_REGRESS_SOCK_DIR: "c:/cirrus/" - DISK_SIZE: 50 - IMAGE_FAMILY: pg-ci-windows-ci - - sysinfo_script: | - chcp - systeminfo - powershell -Command get-psdrive -psprovider filesystem - set - - -task: - name: Windows - Server 2022, VS 2019 - Meson & ninja - << : *WINDOWS_ENVIRONMENT_BASE - - env: - TEST_JOBS: 8 # wild guess, data based value welcome - - # Cirrus defaults to SetErrorMode(SEM_NOGPFAULTERRORBOX | ...). That - # prevents crash reporting from working unless binaries do SetErrorMode() - # themselves. Furthermore, it appears that either python or, more likely, - # the C runtime has a bug where SEM_NOGPFAULTERRORBOX can very - # occasionally *trigger* a crash on process exit - which is hard to debug, - # given that it explicitly prevents crash dumps from working... - # 0x8001 is SEM_FAILCRITICALERRORS | SEM_NOOPENFILEERRORBOX - CIRRUS_WINDOWS_ERROR_MODE: 0x8001 - - MESON_FEATURES: - -Dcpp_args=/std:c++20 - -Dauto_features=disabled - -Dldap=enabled - -Dssl=openssl - -Dtap_tests=enabled - -Dplperl=enabled - -Dplpython=enabled - - <<: *windows_task_template - - depends_on: SanityCheck - only_if: $CI_WINDOWS_ENABLED - - setup_additional_packages_script: | - REM choco install -y --no-progress ... - - setup_hosts_file_script: | - echo 127.0.0.1 pg-loadbalancetest >> c:\Windows\System32\Drivers\etc\hosts - echo 127.0.0.2 pg-loadbalancetest >> c:\Windows\System32\Drivers\etc\hosts - echo 127.0.0.3 pg-loadbalancetest >> c:\Windows\System32\Drivers\etc\hosts - type c:\Windows\System32\Drivers\etc\hosts - - configure_script: | - vcvarsall x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% --buildtype debug -Db_pch=true -Dextra_lib_dirs=c:\openssl\1.1\lib -Dextra_include_dirs=c:\openssl\1.1\include -DTAR=%TAR% %MESON_FEATURES% build - - build_script: | - vcvarsall x64 - ninja -C build %MBUILD_TARGET% - ninja -C build -t missingdeps - - check_world_script: | - vcvarsall x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% - - on_failure: - <<: *on_failure_meson - crashlog_artifacts: - path: "crashlog-*.txt" - type: text/plain - - -task: - << : *WINDOWS_ENVIRONMENT_BASE - name: Windows - Server 2022, MinGW64 - Meson - - # See REPO_CI_AUTOMATIC_TRIGGER_TASKS in .cirrus.star. - trigger_type: $CI_TRIGGER_TYPE_MINGW - - depends_on: SanityCheck - only_if: $CI_MINGW_ENABLED - - env: - TEST_JOBS: 4 # higher concurrency causes occasional failures - CCACHE_DIR: C:/msys64/ccache - CCACHE_MAXSIZE: "500M" - CCACHE_SLOPPINESS: pch_defines,time_macros - CCACHE_DEPEND: 1 - # for some reason mingw plpython cannot find its installation without this - PYTHONHOME: C:/msys64/ucrt64 - # prevents MSYS bash from resetting error mode - MSYS: winjitdebug - # Start bash in current working directory - CHERE_INVOKING: 1 - BASH: C:\msys64\usr\bin\bash.exe -l - - # Keep -Dnls explicitly disabled, as the number of files it creates causes a - # noticeable slowdown. - MESON_FEATURES: >- - -Dnls=disabled - - <<: *windows_task_template - - ccache_cache: - folder: ${CCACHE_DIR} - - setup_additional_packages_script: | - REM C:\msys64\usr\bin\pacman.exe -S --noconfirm ... - - mingw_info_script: | - %BASH% -c "where gcc" - %BASH% -c "gcc --version" - %BASH% -c "where perl" - %BASH% -c "perl --version" - - configure_script: | - %BASH% -c "meson setup %MESON_COMMON_PG_CONFIG_ARGS% -Ddebug=true -Doptimization=g -Db_pch=true %MESON_COMMON_FEATURES% %MESON_FEATURES% -DTAR=%TAR% build" - - build_script: | - %BASH% -c "ninja -C build ${MBUILD_TARGET}" - - upload_caches: ccache - - test_world_script: | - %BASH% -c "meson test %MTEST_ARGS% --num-processes %TEST_JOBS%" - - on_failure: - <<: *on_failure_meson - crashlog_artifacts: - path: "crashlog-*.txt" - type: text/plain - - -task: - name: CompilerWarnings - - # To limit unnecessary work only run this once the SanityCheck - # succeeds. This is particularly important for this task as we intentionally - # use always: to continue after failures. - depends_on: SanityCheck - only_if: $CI_COMPILERWARNINGS_ENABLED - - env: - CPUS: 4 - BUILD_JOBS: 4 - IMAGE_FAMILY: pg-ci-trixie - - # Use larger ccache cache, as this task compiles with multiple compilers / - # flag combinations - CCACHE_MAXSIZE: "1G" - CCACHE_DIR: "/tmp/ccache_dir" - - LINUX_CONFIGURE_FEATURES: *LINUX_CONFIGURE_FEATURES - - <<: *linux_task_template - - sysinfo_script: | - id - uname -a - cat /proc/cmdline - ulimit -a -H && ulimit -a -S - gcc -v - clang -v - export - - ccache_cache: - folder: $CCACHE_DIR - - setup_additional_packages_script: | - #apt-get update - #DEBIAN_FRONTEND=noninteractive apt-get -y install ... - - ### - # Test that code can be built with gcc/clang without warnings - ### - - setup_script: echo "COPT=-Werror" > src/Makefile.custom - - # Trace probes have a history of getting accidentally broken. Use the - # different compilers to build with different combinations of dtrace on/off - # and cassert on/off. - - # gcc, cassert off, dtrace on - always: - gcc_warning_script: | - time ./configure \ - --cache gcc.cache \ - --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ - CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - time make -s -j${BUILD_JOBS} world-bin - - # gcc, cassert on, dtrace off - always: - gcc_a_warning_script: | - time ./configure \ - --cache gcc.cache \ - --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ - CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - time make -s -j${BUILD_JOBS} world-bin - - # clang, cassert off, dtrace off - always: - clang_warning_script: | - time ./configure \ - --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ - CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - time make -s -j${BUILD_JOBS} world-bin - - # clang, cassert on, dtrace on - always: - clang_a_warning_script: | - time ./configure \ - --cache clang.cache \ - --enable-cassert \ - --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ - CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - time make -s -j${BUILD_JOBS} world-bin - - # cross-compile to windows - always: - mingw_cross_warning_script: | - time ./configure \ - --host=x86_64-w64-mingw32ucrt \ - --enable-cassert \ - --without-icu \ - CC="ccache x86_64-w64-mingw32ucrt-gcc" \ - CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - time make -s -j${BUILD_JOBS} world-bin - - ### - # Verify docs can be built - ### - # XXX: Only do this if there have been changes in doc/ since last build - always: - docs_build_script: | - time ./configure \ - --cache gcc.cache \ - CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - time make -s -j${BUILD_JOBS} -C doc - - ### - # Verify headerscheck / cpluspluscheck succeed - # - # - Run both in same script to increase parallelism, use -k to get result of both - # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose - ### - always: - headers_headerscheck_script: | - time ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ - --cache gcc.cache \ - --quiet \ - CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - time make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' - - always: - upload_caches: ccache diff --git a/src/tools/ci/ci_macports_packages.sh b/src/tools/ci/ci_macports_packages.sh index e49f4f703a0..092d8cd7e8f 100755 --- a/src/tools/ci/ci_macports_packages.sh +++ b/src/tools/ci/ci_macports_packages.sh @@ -6,8 +6,7 @@ # when packages are installed or removed. Any package this script is # not instructed to install, will be removed again. # -# This currently expects to be run in a GitHub Actions or cirrus-ci -# macOS environment. +# This currently expects to be run in a GitHub Actions macOS environment. set -e # set -x @@ -39,8 +38,8 @@ fi cache_dmg="macports.hfs.dmg" -if [ "$CIRRUS_CI" != "true" ] && [ "$GITHUB_ACTIONS" != "true" ]; then - echo "expect to be called within cirrus-ci or GitHub Actions" 1>&2 +if [ "$GITHUB_ACTIONS" != "true" ]; then + echo "expect to be called within GitHub Actions" 1>&2 exit 1 fi diff --git a/src/tools/ci/gcp_ram_disk.sh b/src/tools/ci/gcp_ram_disk.sh deleted file mode 100755 index 18dbb2037f5..00000000000 --- a/src/tools/ci/gcp_ram_disk.sh +++ /dev/null @@ -1,27 +0,0 @@ -#!/bin/sh -# Move working directory into a RAM disk for better performance. - -set -e -set -x - -mv $CIRRUS_WORKING_DIR $CIRRUS_WORKING_DIR.orig -mkdir $CIRRUS_WORKING_DIR - -case "`uname`" in - FreeBSD|NetBSD) - mount -t tmpfs tmpfs $CIRRUS_WORKING_DIR - ;; - OpenBSD) - umount /dev/sd0j # unused /usr/obj partition - printf "m j\n\n\nswap\nw\nq\n" | disklabel -E sd0 - swapon /dev/sd0j - # Remove the per-process data segment limit so that mount_mfs can allocate - # large memory filesystems. Without this, mount_mfs mmap() may fail with - # "Cannot allocate memory" if the requested size exceeds the current - # datasize limit. - ulimit -d unlimited - mount -t mfs -o rw,noatime,nodev,-s=10000000 swap $CIRRUS_WORKING_DIR - ;; -esac - -cp -a $CIRRUS_WORKING_DIR.orig/. $CIRRUS_WORKING_DIR/ -- 2.54.0.380.gc69baaf57b
>From 87f09d2c9b7c676c8718d0e00229d7f0c2358919 Mon Sep 17 00:00:00 2001 From: Andres Freund <[email protected]> Date: Thu, 4 Jun 2026 10:17:40 -0400 Subject: [PATCH v12a 3/3] ci: macports improvements I realized two things: 1) We should save the macports cache before building & running tests, we don't want to again start from scratch if the task failed or was cancelled 2) We should use a partial cache match, as it's much faster to start from that, than from scratch. This requires some additional complexity, we now need an explicit restore step and need to check if the install step succeeded. But it's a very substantial improvement in runtime, so it's worthwhile. --- .github/workflows/pg-ci.yml | 24 +++++++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-) diff --git a/.github/workflows/pg-ci.yml b/.github/workflows/pg-ci.yml index dea8d95d729..dd4ff4c91d3 100644 --- a/.github/workflows/pg-ci.yml +++ b/.github/workflows/pg-ci.yml @@ -690,18 +690,23 @@ jobs: sudo sysctl kern.corefile="$HOME/cores/core.%P" - name: "Macports: Compute cache key" - id: mpkey + id: mp-key run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" + # It's faster to start with a partial cache for the same macos + # version than from scratch + echo "restore-key=macports-${macos_major}-" >> "$GITHUB_OUTPUT" - name: "MacPorts: Restore cache" - uses: actions/cache@v5 + id: mp-restore + uses: actions/cache/restore@v5 with: path: ${{ env.MACPORTS_CACHE }} - key: ${{ steps.mpkey.outputs.key }} + key: ${{ steps.mp-key.outputs.key }} + restore-keys: ${{ steps.mp-key.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -714,6 +719,7 @@ jobs: # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - name: "MacPorts: Install dependencies" + id: mp-install env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -727,6 +733,18 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + # Save macports before running build / tests, creating macports can be + # quite slow, so we don't want to start from scratch in the next run. + - name: "MacPorts: Save cache" + uses: actions/cache/save@v5 + # Don't save cache if we had an exact match + if: | + steps.mp-install.conclusion == 'success' && + steps.mp-restore.outputs.cache-hit != 'true' + with: + path: ${{ env.MACPORTS_CACHE }} + key: ${{ steps.mp-key.outputs.key }} + - name: Configure env: PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ -- 2.54.0.380.gc69baaf57b
