Hi,
Attached is an updated version of the CI patches. An example of a test run
with the attached version of this
https://cirrus-ci.com/build/6501998521483264
I again included the commit allowing crash dumps to be collected on windows,
but I don't think it can be merged as-is, and should be left for later.
Changes since v2:
- Address review comments
- Build with further optional features enabled. I think just about everything
reasonable is now enabled on freebsd, linux and macos. There's quite a bit
more that could be done on windows, but I think it's good enough for now.
- I added cross-compilation to windows from linux, to the "warnings"
task. Occasionally there are build-system issues specific to
cross-compilation, and the set of warnings are different.
- Docs are now built as part of the 'CompilerWarnings' task.
- I improved the CI README a bit more, in particular I added docs for the
'ci-os-only' tag I added to the CI logic, which lets one select which
operating systems test get to run on.
- Some of the 'Warnings' tasks now build with --enable-dtrace (once with
optimizations, once without). It's pretty easy to break probes without
seeing the problem locally.
- Switched to using PG_TEST_USE_UNIX_SOCKETS for windows. Without that I was
seeing occasional spurious test failures due to the use of PROVE_FLAGS=
-j10, to make the otherwise serial execution of tests on windows bearable.
- switch macos task to use monterey
- plenty smaller changes / cleanups
There of course is a lot more that can be done [1], but I am pretty happy with
what this covers.
I'd like to commit this soon. There's two aspects that perhaps deserve a bit
more discussion before doing so though:
One I explicitly brought up before:
On 2021-10-01 15:27:52 -0700, Andres Freund wrote:
> One policy discussion that we'd have to have is who should control the images
> used for CI. Right now that's on my personal google cloud account - which I am
> happy to do, but medium - long term that'd not be optimal.
The proposed CI script uses custom images to run linux and freebsd tests. They
are automatically built every day from the repository
https://github.com/anarazel/pg-vm-images/
These images have all the prerequisites pre-installed. For Linux something
similar can be achieved by using dockerfiles referenced the .cirrus.yml file,
but for FreeBSD that's not available. Installing the necessary dependencies on
every run is too time intensive. For linux, the tests run a lot faster in
full-blown VMs than in docker, and full VMs allow a lot more control /
debugging.
I think this may be OK for now, but I also could see arguments for wanting to
transfer the image specifications and the google account to PG properties.
The second attention-worthy point is the choice of a new toplevel ci/
directory as the location for resources referencenced by CI. A few other
projects also use ci/, but I can also see arguments for moving the contents to
e.g. src/tools/ci or such?
Greetings,
Andres Freund
[1] Some ideas for what could make sense to extend CI to in the future:
- also test with msys / mingw on windows
- provide more optional dependencies for windows build
- Extend the set of compiler warnings - as the compiler version is controlled,
we could be more aggressive than we can be via configure.ac.
- Add further distributions / platforms. Possibly as "manual" tasks - the
amount resources one CI user gets is limited, so running tests on all
platforms all the time would make tests take longer. Interesting things
could be:
- further linux distributions, particularly long-term supported ones
- Some of the other BSDs. There currently are no pre-made images for
openbsd/netbsd, but it shouldn't be too hard to script building them.
- running some tests on ARM could be interesting, cirrus supports that for
container based builds now
- run checks like cpluspluscheck as part of the CompilerWarnings task
- add tasks for running tests with tools like asan / ubsan (valgrind will be
too slow).
- consider enable compile-time debugging options like COPY_PARSE_PLAN_TREES,
and run-time force_parallel_mode = regress on some platforms. They seem to
catch a lot of problems during development and are likely affordable enough.
>From 0fdf36d6701b3bb837a62ce8323e37acf7893e5d Mon Sep 17 00:00:00 2001
From: Andres Freund <and...@anarazel.de>
Date: Mon, 15 Mar 2021 09:25:15 -0700
Subject: [PATCH v3 1/2] ci: Add CI for FreeBSD, Linux, MacOS and Windows,
utilizing cirrus-ci.
Author: Andres Freund <and...@anarazel.de>
Author: Thomas Munro <tmu...@postgresql.org>
Author: Melanie Plageman <melanieplage...@gmail.com>
Reviewed-By: Melanie Plageman <melanieplage...@gmail.com>
Discussion: https://postgr.es/m/20211001222752.wrz7erzh4cajv...@alap3.anarazel.de
---
.cirrus.yml | 456 ++++++++++++++++++++++++++++++++
.dockerignore | 7 +
ci/README | 50 ++++
ci/docker/linux_debian_bullseye | 67 +++++
ci/docker/windows_vs_2019 | 122 +++++++++
ci/gcp_freebsd_repartition.sh | 28 ++
ci/pg_ci_base.conf | 14 +
ci/windows_build_config.pl | 13 +
8 files changed, 757 insertions(+)
create mode 100644 .cirrus.yml
create mode 100644 .dockerignore
create mode 100644 ci/README
create mode 100644 ci/docker/linux_debian_bullseye
create mode 100644 ci/docker/windows_vs_2019
create mode 100755 ci/gcp_freebsd_repartition.sh
create mode 100644 ci/pg_ci_base.conf
create mode 100644 ci/windows_build_config.pl
diff --git a/.cirrus.yml b/.cirrus.yml
new file mode 100644
index 00000000000..7a068650206
--- /dev/null
+++ b/.cirrus.yml
@@ -0,0 +1,456 @@
+# What files to preserve in case tests fail
+on_failure: &on_failure
+ log_artifacts:
+ path: "**/**.log"
+ type: text/plain
+ regress_diffs_artifacts:
+ path: "**/**.diffs"
+ type: text/plain
+ tap_artifacts:
+ path: "**/regress_log_*"
+ type: text/plain
+
+env:
+ # accelerate initial clone, but a bit of depth so that concurrent tasks work
+ CIRRUS_CLONE_DEPTH: 100
+ # Useful to be able to analyse what in a script takes long
+ CIRRUS_LOG_TIMESTAMP: true
+ # target to test, for all but windows
+ CHECK: check-world
+ CHECKFLAGS: -Otarget
+ PGCTLTIMEOUT: 120
+ CCACHE_MAXSIZE: "500M"
+ TEMP_CONFIG: ${CIRRUS_WORKING_DIR}/ci/pg_ci_base.conf
+ PG_TEST_EXTRA: kerberos ldap ssl
+
+
+task:
+ name: FreeBSD
+ only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*freebsd.*'
+ compute_engine_instance:
+ image_project: pg-vm-images-aio
+ image: family/pg-aio-freebsd-13-0
+ platform: freebsd
+ cpu: 2
+ memory: 2G
+ disk: 50
+ env:
+ CCACHE_DIR: "/tmp/ccache_dir"
+
+ ccache_cache:
+ folder: "/tmp/ccache_dir"
+ sysinfo_script:
+ - export
+ sysconfig_script:
+ - sudo sysctl kern.corefile='/tmp/%N.%P.core'
+ repartition_script:
+ - ci/gcp_freebsd_repartition.sh
+ create_user_script:
+ - pw useradd postgres
+ - chown -R postgres:postgres .
+ - mkdir -p /tmp/ccache_dir
+ - chown -R postgres:postgres /tmp/ccache_dir
+
+ configure_script: |
+ su postgres -c './configure \
+ --enable-cassert --enable-debug --enable-tap-tests \
+ --enable-nls \
+ \
+ --with-gssapi \
+ --with-icu \
+ --with-ldap \
+ --with-libxml \
+ --with-libxslt \
+ --with-lz4 \
+ --with-pam \
+ --with-perl \
+ --with-python \
+ --with-ssl=openssl \
+ --with-tcl --with-tclconfig=/usr/local/lib/tcl8.6/ \
+ --with-uuid=bsd \
+ \
+ --with-includes=/usr/local/include \
+ --with-libs=/usr/local/lib \
+ \
+ CC="ccache cc" CFLAGS="-O0 -ggdb"'
+ build_script:
+ - su postgres -c 'gmake -s -j3 && gmake -s -j3 -C contrib'
+ upload_caches:
+ - ccache
+
+ # Use of script is to avoid make complaints about fcntl()
+ # https://savannah.gnu.org/bugs/?60774
+ test_world_script:
+ - su postgres -c 'time script test.log gmake -s -j2 ${CHECK} ${CHECKFLAGS}'
+
+ on_failure:
+ <<: *on_failure
+ cores_script: |
+ for corefile in $(find /tmp -name '*.core' 2>/dev/null) ; do
+ binary=$(gdb -quiet -core $corefile -batch -ex 'info auxv' | grep AT_EXECPATH | perl -pe "s/^.*\"(.*)\"\$/\$1/g") ;
+ echo dumping $corefile for $binary ;
+ gdb --batch --quiet -ex "thread apply all bt full" -ex "quit" $binary $corefile;
+ done
+
+
+task:
+ name: Linux
+ only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*linux.*'
+ compute_engine_instance:
+ image_project: pg-vm-images-aio
+ image: family/pg-aio-bullseye
+ platform: linux
+ cpu: 4
+ memory: 2G
+ nested_virtualization: false
+ env:
+ CCACHE_DIR: "/tmp/ccache_dir"
+ DEBUGINFOD_URLS: "https://debuginfod.debian.net";
+
+ ccache_cache:
+ folder: "/tmp/ccache_dir"
+
+ sysinfo_script:
+ - id
+ - uname -a
+ - cat /proc/cmdline
+ - lsblk
+ - ulimit -a -H
+ - ulimit -a -S
+ - export
+ sysconfig_script:
+ - useradd -m postgres
+ - chown -R postgres:postgres .
+ - mkdir -p /tmp/ccache_dir
+ - chown -R postgres:postgres /tmp/ccache_dir
+ - echo '* - memlock 134217728' > /etc/security/limits.d/postgres.conf
+ - su postgres -c 'ulimit -l -H'
+ - su postgres -c 'ulimit -l -S'
+ - echo '/tmp/%e-%s-%p.core' > /proc/sys/kernel/core_pattern
+
+ configure_script: |
+ su postgres -c './configure \
+ --enable-cassert --enable-debug --enable-tap-tests \
+ --enable-nls \
+ \
+ --with-gssapi \
+ --with-icu \
+ --with-ldap \
+ --with-libxml \
+ --with-libxslt \
+ --with-llvm \
+ --with-lz4 \
+ --with-pam \
+ --with-perl \
+ --with-python \
+ --with-selinux \
+ --with-ssl=openssl \
+ --with-systemd \
+ --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ \
+ --with-uuid=e2fs \
+ \
+ CC="ccache gcc" CFLAGS="-O0 -ggdb" \
+ CXX="ccache g++" CXXFLAGS="-O0 -ggdb" \
+ CLANG="ccache clang" '
+ build_script:
+ - su postgres -c 'make -s -j4 && make -s -j4 -C contrib'
+ upload_caches:
+ - ccache
+
+ test_world_script: |
+ su postgres -c '\
+ ulimit -c unlimited; \
+ make -s ${CHECK} ${CHECKFLAGS} -j8 \
+ '
+
+ on_failure:
+ <<: *on_failure
+ cores_script: |
+ for corefile in $(find /tmp -name '*.core' 2>/dev/null) ; do
+ binary=$(gdb -quiet -core $corefile -batch -ex 'info auxv' | grep AT_EXECFN | perl -pe "s/^.*\"(.*)\"\$/\$1/g") ;
+ echo dumping $corefile for $binary ;
+ gdb --batch --quiet -ex "thread apply all bt full" -ex "quit" $binary $corefile ;
+ done
+
+
+task:
+ name: macOS
+ only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*(macos|darwin|osx).*'
+ osx_instance:
+ image: monterey-base
+ env:
+ CIRRUS_WORKING_DIR: ${HOME}/pgsql/
+ CCACHE_DIR: ${HOME}/ccache
+ HOMEBREW_CACHE: ${HOME}/homebrew-cache
+ PERL5LIB: ${HOME}/perl5/lib/perl5
+
+ sysinfo_script:
+ - id
+ - export
+ ccache_cache:
+ folder: ${CCACHE_DIR}
+ homebrew_cache:
+ folder: ${HOMEBREW_CACHE}
+ perl_cache:
+ folder: ~/perl5
+
+ cpan_install_script:
+ - perl -mIPC::Run -e 1 || cpan -T IPC::Run
+ - perl -mIO::Pty -e 1 || cpan -T IO::Pty
+ upload_caches:
+ - perl
+ core_install_script:
+ - sudo chmod 777 /cores
+
+ # XXX: Could we instead install homebrew into a cached directory? The
+ # homebrew installation takes a good bit of time every time, even if the
+ # packages do not need to be downloaded.
+ homebrew_install_script: |
+ brew install \
+ ccache \
+ coreutils \
+ icu4c \
+ krb5 \
+ llvm \
+ lz4 \
+ make \
+ openldap \
+ openssl \
+ python@3.10 \
+ tcl-tk
+
+ brew cleanup -s
+ upload_caches:
+ - homebrew
+
+ configure_script: |
+ LIBS="/usr/local/lib:$LIBS"
+ INCLUDES="/usr/local/include:$INCLUDES"
+
+ PKG_CONFIG_PATH="/usr/local/opt/icu4c/lib/pkgconfig:$PKG_CONFIG_PATH"
+ INCLUDES="/usr/local/opt/icu4c/include:$INCLUDES"
+ LIBS="/usr/local/opt/icu4c/lib:$LIBS"
+
+ PKG_CONFIG_PATH="/usr/local/opt/krb5/lib/pkgconfig:$PKG_CONFIG_PATH"
+ INCLUDES="/usr/local/opt/krb5/include:$INCLUDES"
+ LIBS="/usr/local/opt/krb5/lib:$LIBS"
+
+ LIBS="/usr/local/opt/openldap/lib:$LIBS"
+ INCLUDES="/usr/local/opt/openldap/include:$INCLUDES"
+
+ INCLUDES="/usr/local/opt/openssl/include:$INCLUDES"
+ LIBS="/usr/local/opt/openssl/lib:$LIBS"
+
+ export PKG_CONFIG_PATH
+
+ ./configure \
+ --enable-cassert --enable-debug --enable-tap-tests \
+ --enable-nls \
+ \
+ --with-bonjour \
+ --with-gssapi \
+ --with-icu \
+ --with-ldap \
+ --with-libxml \
+ --with-libxslt \
+ --with-lz4 \
+ --with-perl \
+ --with-python \
+ --with-ssl=openssl \
+ --with-tcl --with-tclconfig=/usr/local/opt/tcl-tk/lib/ \
+ --with-uuid=e2fs \
+ \
+ --prefix=$HOME/install \
+ --with-includes="$INCLUDES" \
+ --with-libs="$LIBS" \
+ \
+ CC="ccache cc" CFLAGS="-O0 -ggdb" \
+ CXX="ccache c++" CXXFLAGS="-O0 -ggdb" \
+ CLANG="ccache /usr/local/opt/llvm/bin/ccache" \
+ LLVM_CONFIG=/usr/local/opt/llvm/bin/llvm-config \
+ PYTHON=/usr/local/opt/python@3.10/bin/python3
+ build_script:
+ - gmake -s -j12 && gmake -s -j12 -C contrib
+ upload_caches:
+ - ccache
+
+ test_world_script:
+ - ulimit -c unlimited
+ - ulimit -n 1024
+ # See freebsd use of script
+ - script test.log gmake -s -j12 ${CHECK} ${CHECKFLAGS}
+
+ on_failure:
+ <<: *on_failure
+ cores_script: |
+ for corefile in $(find /cores/ -name 'core.*' 2>/dev/null) ; do
+ lldb -c $corefile --batch -o 'thread backtrace all' -o 'quit' ;
+ done
+
+
+task:
+ name: Windows
+ only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*windows.*'
+ windows_container:
+ dockerfile: ci/docker/windows_vs_2019
+ cpu: 4
+ memory: 4G
+ env:
+ PROVE_FLAGS: -j10
+ # The default cirrus working dir is in a directory msbuild complains about
+ CIRRUS_WORKING_DIR: "c:/cirrus"
+ # Avoid re-installing over and over
+ NO_TEMP_INSTALL: 1
+ # git's tar doesn't deal with drive letters, see
+ # https://postgr.es/m/b6782dc3-a7b0-ed56-175f-f8f54cb08d67%40dunslane.net
+ TAR: "c:/windows/system32/tar.exe"
+ # Avoids port conflicts
+ PG_TEST_USE_UNIX_SOCKETS: 1
+ PG_REGRESS_SOCK_DIR: "c:/cirrus/"
+
+ sysinfo_script:
+ - chcp
+ - systeminfo
+ - powershell -Command get-psdrive -psprovider filesystem
+ - ps: Get-Item -Path 'HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug'
+ - set
+
+ configure_script:
+ - copy ci\windows_build_config.pl src\tools\msvc\config.pl
+ - vcvarsall x64
+ - perl src/tools/msvc/mkvcbuild.pl
+ build_script:
+ - vcvarsall x64
+ # Disable file tracker, we're never going to rebuild...
+ - msbuild -m /p:TrackFileAccess=false pgsql.sln
+ tempinstall_script:
+ # Installation on windows currently only completely works from src\tools\msvc
+ - cd src\tools\msvc && perl .\install.pl %CIRRUS_WORKING_DIR%\tmp_install
+
+ test_check_parallel_script:
+ - perl src/tools/msvc/vcregress.pl check parallel
+ startcreate_script:
+ - tmp_install\bin\pg_ctl.exe initdb -D tmp_check\db -l tmp_check\initdb.log
+ - echo include '%TEMP_CONFIG%' >> tmp_check\db\postgresql.conf
+ - tmp_install\bin\pg_ctl.exe start -D tmp_check\db -l tmp_check\postmaster.log
+ test_plcheck_script:
+ - perl src/tools/msvc/vcregress.pl plcheck
+ test_isolationcheck_script:
+ - perl src/tools/msvc/vcregress.pl isolationcheck
+ test_modulescheck_script:
+ - perl src/tools/msvc/vcregress.pl modulescheck
+ test_contribcheck_script:
+ - perl src/tools/msvc/vcregress.pl contribcheck
+ stop_script:
+ - tmp_install\bin\pg_ctl.exe stop -D tmp_check\db -l tmp_check\postmaster.log
+ test_ssl_script:
+ - set with_ssl=openssl
+ - perl src/tools/msvc/vcregress.pl taptest .\src\test\ssl\
+ test_subscriptioncheck_script:
+ - perl src/tools/msvc/vcregress.pl taptest .\src\test\subscription\
+ test_authentication_script:
+ - perl src/tools/msvc/vcregress.pl taptest .\src\test\authentication\
+ test_recoverycheck_script:
+ - perl src/tools/msvc/vcregress.pl recoverycheck
+ test_bincheck_script:
+ - perl src/tools/msvc/vcregress.pl bincheck
+ test_upgradecheck_script:
+ - perl src/tools/msvc/vcregress.pl upgradecheck
+ test_ecpgcheck_script:
+ # tries to build additional stuff
+ - vcvarsall x64
+ # References ecpg_regression.proj in the current dir
+ - cd src\tools\msvc
+ - perl vcregress.pl ecpgcheck
+
+ on_failure: *on_failure
+
+
+task:
+ name: CompilerWarnings
+ depends_on:
+ - Linux
+ # task that did not run count as a success, so we need to recheck Linux'
+ # condition here :/
+ only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*linux.*'
+ container:
+ dockerfile: ci/docker/linux_debian_bullseye
+ env:
+ # Use larger ccache cache, as this task compiles with multiple compilers /
+ # flag combinations
+ CCACHE_SIZE: "4GB"
+ CCACHE_DIR: "/tmp/ccache_dir"
+ ccache_cache:
+ folder: "/tmp/ccache_dir"
+
+ sysinfo_script:
+ - gcc -v
+ - clang -v
+
+ setup_script:
+ - echo "COPT=-Werror" > src/Makefile.custom
+
+ ###
+ # Test that code can be built with gcc/clang without warnings
+ ###
+
+ # Trace probes have a history of getting accidentally broken. Use the
+ # different compilers to build with different combinations of dtrace on/off
+ # and cassert on/off.
+
+ # gcc, cassert off, dtrace on
+ always:
+ gcc_warning_script: |
+ time ./configure \
+ --cache gcc.cache CC="ccache gcc" \
+ --enable-dtrace
+ make -s -j4 clean
+ time make -s -j4 world-bin
+ # gcc, cassert on, dtrace off
+ always:
+ gcc_a_warning_script: |
+ time ./configure \
+ --cache gcc.cache CC="ccache gcc" \
+ --enable-cassert
+ make -s -j4 clean
+ time make -s -j4 world-bin
+ # clang, cassert off, dtrace off
+ always:
+ clang_warning_script: |
+ time ./configure \
+ --cache clang.cache CC="ccache clang"
+ make -s -j4 clean
+ time make -s -j4 world-bin
+ # clang, cassert on, dtrace on
+ always:
+ clang_a_warning_script: |
+ time ./configure \
+ --cache clang.cache CC="ccache clang" \
+ --enable-dtrace \
+ --enable-cassert
+ make -s -j4 clean
+ time make -s -j4 world-bin
+ # cross-compile to windows
+ always:
+ mingw_cross_warning_script: |
+ time ./configure \
+ --host=x86_64-w64-mingw32 \
+ --enable-cassert \
+ CC="ccache x86_64-w64-mingw32-gcc"
+ make -s -j4 clean
+ time make -s -j4 world-bin
+
+ ###
+ # Verify docs can be built
+ ###
+
+ # XXX: Only do this if there have been changes in doc/ since last build
+ always:
+ docs_build_script: |
+ time ./configure \
+ --cache gcc.cache CC="ccache gcc"
+ time make -s -j4 -C doc
+
+ always:
+ upload_caches:
+ - ccache
diff --git a/.dockerignore b/.dockerignore
new file mode 100644
index 00000000000..f27b14d5c26
--- /dev/null
+++ b/.dockerignore
@@ -0,0 +1,7 @@
+# Ignore everything, except ci/
+#
+
+# cirrus-ci uses the whole repository as context when building dockerfiles,
+# and the dockerfiles should not reference anything outside of ci/
+*
+!ci/*
diff --git a/ci/README b/ci/README
new file mode 100644
index 00000000000..fda6eb39fb0
--- /dev/null
+++ b/ci/README
@@ -0,0 +1,50 @@
+Postgres Continuous Integration (CI)
+====================================
+
+Postgres has two forms of CI:
+
+1) All supported branches in the main postgres repository are continuously
+ tested via the the the buildfarm. As this covers only the main repository,
+ it cannot be used during development of features.
+
+ For details see https://buildfarm.postgresql.org/
+
+2) For not yet merged development work, CI can be enabled for some git hosting
+ providers. This allows to test patches on a number of platforms before they
+ are merged (or even submitted).
+
+
+Configuring CI on personal repositories
+=======================================
+
+Currently postgres contains CI support utilizing cirrus-ci. cirrus-ci
+currently is only available for github.
+
+
+Enabling cirrus-ci in a github repository
+=========================================
+
+To enable cirrus-ci on a repository, go to
+https://github.com/marketplace/cirrus-ci and select "Public
+Repositories". Then "Install it for free" and "Complete order". The next page
+allows to configure which repositories cirrus-ci has access too. Choose the
+relevant repository and "Install".
+
+See also https://cirrus-ci.org/guide/quick-start/
+
+Once enabled on a repository, future commits and pull-requests in that
+repository will automatically trigger CI builds. These are visible from the
+commit history / PRs, and can also be viewed in the cirrus-ci UI at
+https://cirrus-ci.com/github/<username>/<reponame>/
+
+
+Controlling CI via commit messages
+==================================
+
+The behavior of CI can be controlled by special content in commit
+messages. Currently the following controls are available:
+
+- ci-os-only: {(freebsd|linux|macos|windows)}
+
+ Only runs CI on operating systems specified. This can be useful when
+ addressing portability issues affecting only a subset of platforms.
diff --git a/ci/docker/linux_debian_bullseye b/ci/docker/linux_debian_bullseye
new file mode 100644
index 00000000000..bf48509c9e4
--- /dev/null
+++ b/ci/docker/linux_debian_bullseye
@@ -0,0 +1,67 @@
+FROM debian:bullseye
+RUN \
+ export DEBIAN_FRONTEND=noninteractive && \
+ apt-get -y update && \
+ apt-get -y upgrade --no-install-recommends && \
+ apt-get -y install --no-install-recommends \
+ \
+ build-essential \
+ gdb \
+ make \
+ git \
+ meson \
+ perl \
+ pkg-config \
+ \
+ bison \
+ ccache \
+ clang \
+ flex \
+ g++ \
+ gcc \
+ gettext \
+ \
+ libio-pty-perl \
+ libipc-run-perl \
+ python3-distutils \
+ \
+ libicu-dev \
+ libkrb5-*-heimdal \
+ libkrb5-dev \
+ libldap2-dev \
+ liblz4-dev \
+ libossp-uuid-dev \
+ libpam-dev \
+ libperl-dev \
+ libpython3-dev \
+ libreadline-dev \
+ libselinux*-dev \
+ libssl-dev \
+ libsystemd-dev \
+ liburing-dev \
+ libxml2-dev \
+ libxslt1-dev \
+ llvm-dev \
+ python3-dev \
+ systemtap-sdt-dev \
+ tcl-dev \
+ uuid-dev \
+ \
+ docbook-xml \
+ docbook-xsl \
+ fop \
+ libxml2-utils \
+ xsltproc \
+ \
+ krb5-admin-server \
+ krb5-kdc \
+ krb5-user \
+ ldap-utils \
+ locales-all \
+ slapd \
+ \
+ g++-mingw-w64-x86-64-win32 \
+ gcc-mingw-w64-x86-64-win32 \
+ libz-mingw-w64-dev \
+ && \
+ apt-get clean && rm -rf /var/lib/apt/lists/*
diff --git a/ci/docker/windows_vs_2019 b/ci/docker/windows_vs_2019
new file mode 100644
index 00000000000..5e0ff91ab6e
--- /dev/null
+++ b/ci/docker/windows_vs_2019
@@ -0,0 +1,122 @@
+# escape=`
+
+# We used to use the visual studio container, but it's too outdated (&bloated) now
+FROM cirrusci/windowsservercore:2019
+
+SHELL ["powershell", "-NoLogo", "-NoProfile", "-Command"]
+
+
+# Install commandline debugger and log all crashes to c:\cirrus\crashlog.txt
+#
+# Done manually as doing this via chocolatey / the installer directly, ends up
+# with a lot of unnecessary chaff, making the layer unnecessarily large.
+RUN `
+ mkdir c:\t ; `
+ cd c:\t ; `
+ `
+ setx PATH \"C:\Windows Kits\10\Debuggers\x64;$Env:PATH\" /m ; `
+ `
+ curl.exe -sSL -o 'windsdksetup.exe' https://download.microsoft.com/download/9/7/9/97982c1d-d687-41be-9dd3-6d01e52ceb68/windowssdk/winsdksetup.exe ; `
+ echo 'starting sdk installation (for debugger)' ; `
+ Start-Process -Wait -FilePath ".\windsdksetup.exe" `
+ -ArgumentList '/Features OptionId.WindowsDesktopDebuggers /layout c:\t\sdk /quiet /norestart /log c:\t\sdk.log' `
+ ; `
+ `
+ Start-Process -Wait -FilePath msiexec.exe `
+ -ArgumentList '/a \"C:\t\sdk\Installers\X64 Debuggers And Tools-x64_en-us.msi\" /qb /log install2.log' `
+ ; `
+ C:\Windows` Kits\10\Debuggers\x64\cdb.exe -version ; `
+ `
+ Set-ItemProperty -Path 'HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug' -Name 'Debugger' -Value '\"C:\Windows Kits\10\Debuggers\x64\cdb.exe\" -p %ld -e %ld -g -kqm -c \".lines -e; .symfix+ ;.logappend c:\cirrus\crashlog.txt ; !peb; ~*kP ; .logclose ; q \"' ; `
+ New-ItemProperty -Path 'HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug' -Name 'Auto' -Value 1 -PropertyType DWord ; `
+ Get-ItemProperty -Path 'HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug' -Name Debugger ; `
+ `
+ cd c:\ ; `
+ Remove-Item C:\t -Force -Recurse
+
+
+# Install perl, python, flex and bison.
+#
+# Done manually as choco takes a lot longer. I think it's download issues with
+# powershell's download stuff? That's why curl.exe is directly used here at
+# least...
+#
+# Using perl 5.26.3.1 for now, as newer versions don't currently work
+# correctly for plperl
+RUN `
+ mkdir c:\t ; `
+ cd c:\t ; `
+ `
+ echo 'adding to path, before setup below, so changes are not overwritten' ; `
+ setx PATH \"C:\strawberry\perl\bin;C:\winflexbison;C:\Program Files\Git\usr\bin;$Env:PATH\" /m ; `
+ `
+ curl.exe -sSL -o perl.zip `
+ https://strawberryperl.com/download/5.26.3.1/strawberry-perl-5.26.3.1-64bit-portable.zip ; `
+ echo 'installing perl' ; `
+ 7z.exe x .\perl.zip -xr!c -oc:\strawberry ; `
+ `
+ curl.exe -sSL -o python.exe https://www.python.org/ftp/python/3.10.0/python-3.10.0-amd64.exe ; `
+ echo 'installing python' ; `
+ Start-Process -Wait -FilePath ".\python.exe" `
+ -ArgumentList `
+ '/quiet', 'SimpleInstall=1', 'PrependPath=1', 'CompileAll=1', `
+ 'TargetDir=c:\python\', 'InstallAllUsers=1', 'Shortcuts=0', `
+ 'Include_docs=0', 'Include_tcltk=0', 'Include_tests=0' `
+ ; `
+ `
+ curl.exe -sSL -o winflexbison.zip `
+ https://github.com/lexxmark/winflexbison/releases/download/v2.5.24/win_flex_bison-2.5.24.zip ; `
+ echo 'installing winflexbison' ; `
+ 7z.exe x .\winflexbison.zip -oc:\winflexbison ; `
+ Rename-Item -Path c:\winflexbison\win_flex.exe c:\winflexbison\flex.exe ; `
+ Rename-Item -Path c:\winflexbison\win_bison.exe c:\winflexbison\bison.exe ; `
+ `
+ cd c:\ ; `
+ Remove-Item C:\t -Force -Recurse
+
+
+# Install openssl
+#
+# This is one of the windows binararies referenced by the openssl wiki. See
+# https://www.openssl.org/community/binaries.html and
+# https://wiki.openssl.org/index.php/Binaries
+#
+# It might be nicer to switch to the openssl built as part of curl-for-win,
+# but recent releases only build openssl 3, and that still seems troublesome
+# on windows,
+RUN `
+ mkdir c:\t ; `
+ cd c:\t ; `
+ `
+ curl.exe -o openssl-setup.exe -sSL https://slproweb.com/download/Win64OpenSSL-1_1_1L.exe ; `
+ echo 'staring openssl installation' ; `
+ Start-Process -Wait -FilePath ".\openssl-setup.exe" `
+ -ArgumentList '/DIR=c:\openssl\1.1.1l\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' ; `
+ `
+ cd c:\ ; `
+ Remove-Item C:\t -Force -Recurse
+
+
+# Install visual studio
+#
+# Adding path to vcvarsall.bat to PATH so user of container dont't need to
+# know the full path
+RUN `
+ mkdir c:\t ; `
+ cd c:\t ; `
+ setx PATH \"c:\BuildTools\VC\Auxiliary\Build;$Env:PATH\" /m ; `
+ `
+ curl.exe -sSL -o c:\t\vs_buildtools.exe https://aka.ms/vs/16/release/vs_buildtools.exe ; `
+ echo 'starting visual studio installation' ; `
+ Start-Process -Wait `
+ -FilePath c:\t\vs_buildtools.exe `
+ -ArgumentList `
+ '--quiet', '--wait', '--norestart', '--nocache', `
+ '--installPath', 'c:\BuildTools', `
+ '--add', 'Microsoft.VisualStudio.Component.VC.Tools.x86.x64', `
+ '--add', 'Microsoft.VisualStudio.Component.Windows10SDK.20348' ; `
+ `
+ cd c:\ ; `
+ Remove-Item C:\t -Force -Recurse ; `
+ Remove-Item -Force -Recurse ${Env:TEMP}\*; `
+ Remove-Item -Force -Recurse \"${Env:ProgramData}\Package Cache\"
diff --git a/ci/gcp_freebsd_repartition.sh b/ci/gcp_freebsd_repartition.sh
new file mode 100755
index 00000000000..2d5e1738998
--- /dev/null
+++ b/ci/gcp_freebsd_repartition.sh
@@ -0,0 +1,28 @@
+#!/bin/sh
+
+set -e
+set -x
+
+# The default filesystem on freebsd gcp images is very slow to run tests on,
+# due to its 32KB block size
+#
+# XXX: It'd probably better to fix this in the image, using something like
+# https://people.freebsd.org/~lidl/blog/re-root.html
+
+# fix backup partition table after resize
+gpart recover da0
+gpart show da0
+# kill swap, so we can delete a partition
+swapoff -a || true
+# (apparently we can only have 4!?)
+gpart delete -i 3 da0
+gpart add -t freebsd-ufs -l data8k -a 4096 da0
+gpart show da0
+newfs -U -b 8192 /dev/da0p3
+
+# Migrate working directory
+du -hs $CIRRUS_WORKING_DIR
+mv $CIRRUS_WORKING_DIR $CIRRUS_WORKING_DIR.orig
+mkdir $CIRRUS_WORKING_DIR
+mount -o noatime /dev/da0p3 $CIRRUS_WORKING_DIR
+cp -r $CIRRUS_WORKING_DIR.orig/* $CIRRUS_WORKING_DIR/
diff --git a/ci/pg_ci_base.conf b/ci/pg_ci_base.conf
new file mode 100644
index 00000000000..d8faa9c26c1
--- /dev/null
+++ b/ci/pg_ci_base.conf
@@ -0,0 +1,14 @@
+# Tends to produce too many core files, taking a long time
+restart_after_crash = false
+
+# So that tests using the "manually" started postgres on windows can use
+# prepared statements
+max_prepared_transactions = 10
+
+# Settings that make logs more useful
+log_autovacuum_min_duration = 0
+log_checkpoints = true
+log_connections = true
+log_disconnections = true
+log_line_prefix = '%m [%p][%b] %q[%a][%v:%x] '
+log_lock_waits = true
diff --git a/ci/windows_build_config.pl b/ci/windows_build_config.pl
new file mode 100644
index 00000000000..ba82b13d69a
--- /dev/null
+++ b/ci/windows_build_config.pl
@@ -0,0 +1,13 @@
+use strict;
+use warnings;
+
+our $config;
+
+$config->{"tap_tests"} = 1;
+$config->{"asserts"} = 1;
+
+$config->{"openssl"} = "c:/openssl/1.1.1l/";
+$config->{"perl"} = "c:/strawberry/perl/";
+$config->{"python"} = "c:/python/";
+
+1;
--
2.34.0
>From 6dfb5cba7579878b147ab9e3c9f3cf23da947b39 Mon Sep 17 00:00:00 2001
From: Andres Freund <and...@anarazel.de>
Date: Thu, 9 Sep 2021 17:49:39 -0700
Subject: [PATCH v3 2/2] windows: Improve crash / assert / exception handling.
---
src/backend/main/main.c | 45 +++++++++++++++++++++++++++++++++++++++--
.cirrus.yml | 7 +++++++
2 files changed, 50 insertions(+), 2 deletions(-)
diff --git a/src/backend/main/main.c b/src/backend/main/main.c
index ad84a45e28c..5115ad91898 100644
--- a/src/backend/main/main.c
+++ b/src/backend/main/main.c
@@ -26,6 +26,10 @@
#include <sys/param.h>
#endif
+#if defined(WIN32)
+#include <crtdbg.h>
+#endif
+
#if defined(_M_AMD64) && _MSC_VER == 1800
#include <math.h>
#include <versionhelpers.h>
@@ -237,8 +241,45 @@ startup_hacks(const char *progname)
exit(1);
}
- /* In case of general protection fault, don't show GUI popup box */
- SetErrorMode(SEM_FAILCRITICALERRORS | SEM_NOGPFAULTERRORBOX);
+ /*
+ * SEM_FAILCRITICALERRORS causes more errors to be reported to
+ * callers.
+ *
+ * We used to also specify SEM_NOGPFAULTERRORBOX, but that prevents
+ * windows crash reporting from working. Which includes registered
+ * just-in-time debuggers. Now we try to disable sources of popups
+ * separately below (note that SEM_NOGPFAULTERRORBOX didn't actually
+ * prevent several sources of popups).
+ */
+ SetErrorMode(SEM_FAILCRITICALERRORS);
+
+ /*
+ * Show errors on stderr instead of popup box (note this doesn't
+ * affect errors originating in the C runtime, see below).
+ */
+ _set_error_mode(_OUT_TO_STDERR);
+
+
+ /*
+ * By default abort() only generates a crash-dump in *non* debug
+ * builds. As our Assert() / ExceptionalCondition() uses abort(),
+ * leaving the default in place would make debugging harder.
+ */
+#ifndef __MINGW64__
+ _set_abort_behavior(_CALL_REPORTFAULT | _WRITE_ABORT_MSG,
+ _CALL_REPORTFAULT | _WRITE_ABORT_MSG);
+#endif
+
+ /*
+ * In case of errors (including assertions) on the C runtime level,
+ * report them to stderr (and the debugger) instead of displaying a
+ * popup. This is C runtime specific and thus the above incantations
+ * aren't sufficient to suppress these popups.
+ */
+ _CrtSetReportMode(_CRT_ASSERT, _CRTDBG_MODE_FILE | _CRTDBG_MODE_DEBUG);
+ _CrtSetReportMode(_CRT_ERROR, _CRTDBG_MODE_FILE | _CRTDBG_MODE_DEBUG);
+ _CrtSetReportFile(_CRT_ASSERT, _CRTDBG_FILE_STDERR);
+ _CrtSetReportFile(_CRT_ERROR, _CRTDBG_FILE_STDERR);
#if defined(_M_AMD64) && _MSC_VER == 1800
diff --git a/.cirrus.yml b/.cirrus.yml
index 7a068650206..90c9c802d80 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -363,6 +363,13 @@ task:
- cd src\tools\msvc
- perl vcregress.pl ecpgcheck
+ always:
+ cores_script:
+ - cat crashlog.txt || true
+ dump_artifacts:
+ path: "crashlog.txt"
+ type: text/plain
+
on_failure: *on_failure
--
2.34.0
