On Fri, Jul 29, 2022 at 10:04:04AM +1200, Thomas Munro wrote: > > > XXX We would never want this to run by default in CI, but it'd be nice > > > to be able to ask for it with ci-os-only! (See commented out line) > > > only_if: $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*cygwin.*' > > > > Doesn't this already do what's needed? > > As long as it doesn't also check: CHANGE_MESSAGE !~ 'ci-os-only', > > the task will runs only on request. > > Yeah I was just trying to say that I was sharing the script in a way > that always runs, but for commit we'd want that.
That makes more sense after noticing that you created a cf entry (for which cfbot has been skipping my patch due to my "only_if" line). There's still a few persistent issues: This fails ~50% of the time in recovery 010-truncate I hacked around this by setting data_sync_retry. https://cirrus-ci.com/task/5289444063313920 I found these, not sure if they're relevant. https://www.postgresql.org/message-id/flat/CAA4eK1Kft05mwNuZbTVRmz8SNS3r%2BuriuCT8DxL5KJy5btoS-A%40mail.gmail.com https://www.postgresql.org/message-id/flat/CAFiTN-uGxgo5258hZy2QJoz%3Ds7_Cs7v9%3Db8Z2GgFV7qmQUOwxw%40mail.gmail.com And an fsync abort in 013 which seems similar to this other one. data_sync_retry also avoids this issue. https://cirrus-ci.com/task/6283023745286144?logs=cores#L34 https://www.postgresql.org/message-id/flat/CAMVYW_4QhjZ-19Xpr2x1B19soRCNu1BXHM8g1mOnAVtd5VViDw%40mail.gmail.com And sometimes various assertions failing in regress parallel_select (and then times out) https://api.cirrus-ci.com/v1/artifact/task/5537540282253312/log/src/test/regress/log/postmaster.log https://api.cirrus-ci.com/v1/artifact/task/6108746773430272/log/src/test/regress/log/postmaster.log Or "could not map dynamic shared memory segment" (actually in 027-stream-regress): https://cirrus-ci.com/task/6168860746317824 And segfault in vacuum parallel https://api.cirrus-ci.com/v1/artifact/task/5404589569605632/log/src/test/regress/log/postmaster.log Sometimes semctl() failed: Resource temporarily unavailable https://api.cirrus-ci.com/v1/artifact/task/5027860623654912/log/src/test/subscription/tmp_check/log/014_binary_publisher.log https://api.cirrus-ci.com/v1/artifact/task/5027860623654912/log/src/bin/pg_rewind/tmp_check/log/001_basic_standby_local.log Some more https://cirrus-ci.com/task/6468927780814848 If you're lucky, there's only 1 or 2 problems, of which those are different symptoms.. Maybe for now this needs to disable tap tests :( This shows that it *can* pass, if slowly, and infrequently: https://cirrus-ci.com/task/6546858536337408 This fixes my changes to configure for getopt. And simplifies the changes to *.pl (the .exe changes weren't necessary at all). And removes the changes for implicit-fallthrough; I realized that configure was just deciding that it didn't work and not using it at all. And adds support for backtraces. And remove kerberos and and add libxml Why did you write "|| exit /b 1" in all the bash invocations ? I think cirrus handles that automatically. -- Justin
>From b929ea7acc33a2fda1ec10693736a2fa83d364e1 Mon Sep 17 00:00:00 2001 From: Thomas Munro <thomas.mu...@gmail.com> Date: Mon, 25 Jul 2022 23:05:10 +1200 Subject: [PATCH v3] WIP CI support for Cygwin. ci-os-only: cygwin See also: d8e78714-dc77-4a64-783f-e863ba4d9...@2ndquadrant.com https://cirrus-ci.com/task/5145086722834432 XXX This should use a canned Docker image with all the right packages installed? But if the larger image is slower to start, then maybe not... --- .cirrus.yml | 67 +++++++++++++++++++++++ configure | 2 +- configure.ac | 2 +- src/test/perl/PostgreSQL/Test/Cluster.pm | 4 +- src/test/perl/PostgreSQL/Test/Utils.pm | 12 +++- src/test/recovery/t/020_archive_status.pl | 2 +- src/tools/ci/cores_backtrace.sh | 28 +++++++++- src/tools/ci/pg_ci_base.conf | 2 + 8 files changed, 109 insertions(+), 10 deletions(-) diff --git a/.cirrus.yml b/.cirrus.yml index 4b7918ef456..84341ac1b94 100644 --- a/.cirrus.yml +++ b/.cirrus.yml @@ -34,6 +34,7 @@ on_failure: &on_failure - "**/*.log" - "**/*.diffs" - "**/regress_log_*" + - "**/*.stackdump" type: text/plain task: @@ -464,6 +465,72 @@ task: type: text/plain +task: + name: Windows - Cygwin + #XXX only_if: $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*cygwin.*' + timeout_in: 90m + + env: + CPUS: 4 + BUILD_JOBS: 4 + TEST_JOBS: 1 + CCACHE_DIR: /tmp/ccache + CONFIGURE_FLAGS: --enable-debug --enable-tap-tests --with-ldap --with-ssl=openssl --with-libxml --enable-cassert + # --with-gssapi + CONFIGURE_CACHE: /tmp/ccache/configure.cache + PG_TEST_USE_UNIX_SOCKETS: 1 + CCACHE_LOGFILE: ccache.log + EXTRA_REGRESS_OPTS: --max-connections=1 + PG_TEST_EXTRA: ldap ssl # disable kerberos + + windows_container: + image: cirrusci/windowsservercore:2019 + os_version: 2019 + cpu: $CPUS + memory: 4G + + setup_additional_packages_script: | + choco install -y --no-progress cygwin + C:\tools\cygwin\cygwinsetup.exe -q -P cygrunsrv,make,gcc-core,ccache,binutils,libtool,pkg-config,flex,bison,zlib-devel,libxml2-devel,libxslt-devel,libssl-devel,openldap-devel,libreadline-devel,perl,perl-IPC-Run + REM libkrb5-devel,krb5-server + C:\tools\cygwin\bin\bash.exe --login -c "cygserver-config -y" + C:\tools\cygwin\bin\bash.exe --login -c "echo 'kern.ipc.semmni 1024' >> /etc/cygserver.conf" + C:\tools\cygwin\bin\bash.exe --login -c "echo 'kern.ipc.semmns 1024' >> /etc/cygserver.conf" + C:\tools\cygwin\bin\bash.exe --login -c "net start cygserver" + + sysinfo_script: | + chcp + systeminfo + powershell -Command get-psdrive -psprovider filesystem + set + C:\tools\cygwin\bin\bash.exe --login -c "id; uname -a; ulimit -a -H; ulimit -a -S; export" + + ccache_cache: + folder: C:\tools\cygwin\tmp\ccache + + configure_script: + # Try to configure with the cache file, and retry without if it fails, in case the flags changed. + - C:\tools\cygwin\bin\bash.exe --login -c "cd '%cd%' && for i in 1 2; do ./configure --cache-file=${CONFIGURE_CACHE} ${CONFIGURE_FLAGS} CC='ccache gcc' CFLAGS='-Og -ggdb' && break; rm -v ${CONFIGURE_CACHE}; done" + + build_script: + #- C:\tools\cygwin\bin\bash.exe --login -c "ccache --max-size ${CCACHE_MAXSIZE}" + - C:\tools\cygwin\bin\bash.exe --login -c "ccache --zero-stats" + - C:\tools\cygwin\bin\bash.exe --login -c "cd '%cd%' && make -s -j ${BUILD_JOBS} world-bin" + - C:\tools\cygwin\bin\bash.exe --login -c "ccache --show-stats" + + upload_caches: ccache + + test_world_script: + #- C:\tools\cygwin\bin\bash.exe --login -c "cd '%cd%' && timeout 44m make -s -j ${TEST_JOBS} check ${CHECKFLAGS} -C src/test/subscription" + #- C:\tools\cygwin\bin\bash.exe --login -c "cd '%cd%' && timeout 44m make -s -j ${TEST_JOBS} check ${CHECKFLAGS} -C src/test/recovery" + #- C:\tools\cygwin\bin\bash.exe --login -c "cd '%cd%' && timeout 44m make -s check ${CHECKFLAGS} -C src/bin -j 2" + - C:\tools\cygwin\bin\bash.exe --login -c "cd '%cd%' && timeout 77m make -s -j ${TEST_JOBS} ${CHECK} PROVE_FLAGS='-j2 --timer' ${CHECKFLAGS}" + + on_failure: + <<: *on_failure + cores_script: + - C:\tools\cygwin\bin\bash.exe --login -c "cd '%cd%' && src/tools/ci/cores_backtrace.sh cygwin ." + task: name: CompilerWarnings diff --git a/configure b/configure index c5bc3823958..d147cf372db 100755 --- a/configure +++ b/configure @@ -17000,7 +17000,7 @@ fi # mingw has adopted a GNU-centric interpretation of optind/optreset, # so always use our version on Windows. -if test "$PORTNAME" = "win32"; then +if test "$PORTNAME" = "win32" -o "$PORTNAME" = "cygwin"; then case " $LIBOBJS " in *" getopt.$ac_objext "* ) ;; *) LIBOBJS="$LIBOBJS getopt.$ac_objext" diff --git a/configure.ac b/configure.ac index 61d0dd5d586..6dba8291d64 100644 --- a/configure.ac +++ b/configure.ac @@ -1947,7 +1947,7 @@ fi # mingw has adopted a GNU-centric interpretation of optind/optreset, # so always use our version on Windows. -if test "$PORTNAME" = "win32"; then +if test "$PORTNAME" = "win32" -o "$PORTNAME" = "cygwin"; then AC_LIBOBJ(getopt) AC_LIBOBJ(getopt_long) fi diff --git a/src/test/perl/PostgreSQL/Test/Cluster.pm b/src/test/perl/PostgreSQL/Test/Cluster.pm index c8c7bc5045a..29894b6a98c 100644 --- a/src/test/perl/PostgreSQL/Test/Cluster.pm +++ b/src/test/perl/PostgreSQL/Test/Cluster.pm @@ -1052,7 +1052,7 @@ sub enable_restoring # the path contains spaces. $path =~ s{\\}{\\\\}g if ($PostgreSQL::Test::Utils::windows_os); my $copy_command = - $PostgreSQL::Test::Utils::windows_os + $PostgreSQL::Test::Utils::windows_os && !$PostgreSQL::Test::Utils::is_cygwin ? qq{copy "$path\\\\%f" "%p"} : qq{cp "$path/%f" "%p"}; @@ -1122,7 +1122,7 @@ sub enable_archiving # the path contains spaces. $path =~ s{\\}{\\\\}g if ($PostgreSQL::Test::Utils::windows_os); my $copy_command = - $PostgreSQL::Test::Utils::windows_os + $PostgreSQL::Test::Utils::windows_os && !$PostgreSQL::Test::Utils::is_cygwin ? qq{copy "%p" "$path\\\\%f"} : qq{cp "%p" "$path/%f"}; diff --git a/src/test/perl/PostgreSQL/Test/Utils.pm b/src/test/perl/PostgreSQL/Test/Utils.pm index 1ca2cc59170..c7786089a4b 100644 --- a/src/test/perl/PostgreSQL/Test/Utils.pm +++ b/src/test/perl/PostgreSQL/Test/Utils.pm @@ -88,10 +88,11 @@ our @EXPORT = qw( $windows_os $is_msys2 + $is_cygwin $use_unix_sockets ); -our ($windows_os, $is_msys2, $use_unix_sockets, $timeout_default, +our ($windows_os, $is_msys2, $is_cygwin, $use_unix_sockets, $timeout_default, $tmp_check, $log_path, $test_logfile); BEGIN @@ -140,13 +141,18 @@ BEGIN $ENV{PGAPPNAME} = basename($0); # Must be set early - $windows_os = $Config{osname} eq 'MSWin32' || $Config{osname} eq 'msys'; + $windows_os = $Config{osname} eq 'MSWin32' || $Config{osname} eq 'msys' || + $Config{osname} eq 'cygwin'; + # Check if this environment is MSYS2. $is_msys2 = $windows_os && -x '/usr/bin/uname' && `uname -or` =~ /^[2-9].*Msys/; + # Check if this environment is Cygwin + $is_cygwin = $Config{osname} eq 'cygwin'; + if ($windows_os) { require Win32API::File; @@ -707,7 +713,7 @@ sub dir_symlink { my $oldname = shift; my $newname = shift; - if ($windows_os) + if ($windows_os && !$is_cygwin) { $oldname =~ s,/,\\,g; $newname =~ s,/,\\,g; diff --git a/src/test/recovery/t/020_archive_status.pl b/src/test/recovery/t/020_archive_status.pl index e6e4eb56a90..0b2716fd7c9 100644 --- a/src/test/recovery/t/020_archive_status.pl +++ b/src/test/recovery/t/020_archive_status.pl @@ -26,7 +26,7 @@ my $primary_data = $primary->data_dir; # a portable solution, use an archive command based on a command known to # work but will fail: copy with an incorrect original path. my $incorrect_command = - $PostgreSQL::Test::Utils::windows_os + $PostgreSQL::Test::Utils::windows_os && !$PostgreSQL::Test::Utils::is_cygwin ? qq{copy "%p_does_not_exist" "%f_does_not_exist"} : qq{cp "%p_does_not_exist" "%f_does_not_exist"}; $primary->safe_psql( diff --git a/src/tools/ci/cores_backtrace.sh b/src/tools/ci/cores_backtrace.sh index 28d3cecfc67..64c980039cd 100755 --- a/src/tools/ci/cores_backtrace.sh +++ b/src/tools/ci/cores_backtrace.sh @@ -1,5 +1,8 @@ #! /bin/sh +#set -e +set -x + if [ $# -ne 2 ]; then echo "cores_backtrace.sh <os> <directory>" exit 1 @@ -8,9 +11,21 @@ fi os=$1 directory=$2 +findargs='' case $os in freebsd|linux|macos) - ;; + ;; + cygwin) + # XXX Evidently I don't know how to write two arguments here without pathname expansion later, other than eval. + #findargs='-name "*.stackdump"' + for corefile in $(find "$directory" -type f -name "*.stackdump") ; do + binary=`basename "$corefile" .stackdump` + echo;echo; + echo "dumping ${corefile} for ${binary}" + awk '/^0/{print $2}' $corefile |addr2line -f -i -e ./src/backend/postgres.exe + done + exit 0 + ;; *) echo "unsupported operating system ${os}" exit 1 @@ -18,7 +33,7 @@ case $os in esac first=1 -for corefile in $(find "$directory" -type f) ; do +for corefile in $(find "$directory" -type f $findargs) ; do if [ "$first" -eq 1 ]; then first=0 else @@ -28,6 +43,13 @@ for corefile in $(find "$directory" -type f) ; do if [ "$os" = 'macos' ]; then lldb -c $corefile --batch -o 'thread backtrace all' -o 'quit' + elif [ "$os" = 'cygwin' ]; then + # https://cirrus-ci.com/task/4964259674193920 + #binary=${corefile%.stackdump} + #binary=${corefile#*/} + binary=`basename "$corefile" .stackdump` + echo "dumping ${corefile} for ${binary}" + awk '/^0/{print $2}' $corefile |addr2line -f -i -e ./src/backend/postgres.exe else auxv=$(gdb --quiet --core ${corefile} --batch -ex 'info auxv' 2>/dev/null) if [ $? -ne 0 ]; then @@ -48,3 +70,5 @@ for corefile in $(find "$directory" -type f) ; do gdb --batch --quiet -ex "thread apply all bt full" -ex "quit" "$binary" "$corefile" 2>/dev/null fi done + +exit 0 diff --git a/src/tools/ci/pg_ci_base.conf b/src/tools/ci/pg_ci_base.conf index d8faa9c26c1..206dd993ccc 100644 --- a/src/tools/ci/pg_ci_base.conf +++ b/src/tools/ci/pg_ci_base.conf @@ -12,3 +12,5 @@ log_connections = true log_disconnections = true log_line_prefix = '%m [%p][%b] %q[%a][%v:%x] ' log_lock_waits = true + +data_sync_retry = on -- 2.17.1
