Hi, On 2022-09-26 14:15:35 -0700, Peter Geoghegan wrote: > On Mon, Sep 26, 2022 at 1:27 PM Andres Freund <and...@anarazel.de> wrote: > > > Some feedback: > > > * I gather that "running" as it appears in commands like "meson test > > > --setup running" refers to a particular setup named "running", that > > > you invented as part of creating a meson-ish substitute for > > > installcheck. Can "running" be renamed to something that makes it > > > obvious that it's a Postgres thing, and not a generic meson thing? > > > > Yes. The only caveat is that it makes lines longer, because it's included in > > the printed test line (there's no real requirement to have the test suite > > and > > the setup named the same,b ut it seems confusing not to) > > Probably doesn't have to be too long. And I'm not sure of the details. > Just a small thing from my point of view.
Attached is an updated version of that patch. I left the name as 'running' because a postgres- or pg- prefix felt to awkward. This just adds fairly minimal documentation for the 'running' setup, while we now have some basic docs for building with meson, we don't yet have a "translation" of regress.sgml. Not sure how to structure that best, either. I plan to commit that soon. This likely isn't the be-all-end-all, but it's quite useful as-is. Greetings, Andres Freund
>From a50cc9ac3045b0ef1cfd3cc76b3ff5759bdb85ef Mon Sep 17 00:00:00 2001 From: Andres Freund <and...@anarazel.de> Date: Tue, 6 Dec 2022 19:24:44 -0800 Subject: [PATCH v2] meson: Add 'running' test setup, as a replacement for installcheck To run all tests that support running against existing server: $ meson test --setup running To run just the main pg_regress tests against existing server: $ meson test --setup running regress-running/regress To ensure the 'running' setup continues to work, test it as part of the freebsd CI task. Discussion: https://postgr.es/m/CAH2-Wz=xdqcmloo7rr_i6fkqddmcyb9q5gstnfuuqxroghb...@mail.gmail.com --- meson.build | 91 ++++++++++++++++--- contrib/basic_archive/meson.build | 3 + contrib/pg_freespacemap/meson.build | 3 + contrib/pg_stat_statements/meson.build | 4 + contrib/pg_walinspect/meson.build | 3 + contrib/test_decoding/meson.build | 5 + src/interfaces/ecpg/test/meson.build | 1 + doc/src/sgml/installation.sgml | 6 ++ .cirrus.yml | 15 +++ src/test/isolation/meson.build | 1 + src/test/modules/commit_ts/meson.build | 3 + src/test/modules/snapshot_too_old/meson.build | 3 + src/test/modules/test_oat_hooks/meson.build | 1 + src/test/modules/test_pg_dump/meson.build | 2 + src/test/modules/test_slru/meson.build | 1 + src/test/modules/worker_spi/meson.build | 4 +- src/test/regress/meson.build | 1 + 17 files changed, 132 insertions(+), 15 deletions(-) diff --git a/meson.build b/meson.build index 39fc3ddab26..3cb50c0b172 100644 --- a/meson.build +++ b/meson.build @@ -2920,6 +2920,20 @@ endif # Test Generation ############################################################### +# When using a meson version understanding exclude_suites, define a +# 'tmp_install' test setup (the default) that excludes tests running against a +# pre-existing install and a 'running' setup that conflicts with creation of +# the temporary installation and tap tests (which don't support running +# against a running server). + +running_suites = [] +install_suites = [] +if meson.version().version_compare('>=0.57') + runningcheck = true +else + runningcheck = false +endif + testwrap = files('src/tools/testwrap') foreach test_dir : tests @@ -2927,7 +2941,6 @@ foreach test_dir : tests testwrap, '--basedir', meson.build_root(), '--srcdir', test_dir['sd'], - '--testgroup', test_dir['name'], ] foreach kind, v : test_dir @@ -2940,55 +2953,94 @@ foreach test_dir : tests if kind in ['regress', 'isolation', 'ecpg'] if kind == 'regress' runner = pg_regress + fallback_dbname = 'regression_@0@' elif kind == 'isolation' runner = pg_isolation_regress + fallback_dbname = 'isolation_regression_@0@' elif kind == 'ecpg' runner = pg_regress_ecpg + fallback_dbname = 'ecpg_regression_@0@' endif - test_output = test_result_dir / test_dir['name'] / kind + test_group = test_dir['name'] + test_group_running = test_dir['name'] + '-running' - test_command = [ + test_output = test_result_dir / test_group / kind + test_output_running = test_result_dir / test_group_running/ kind + + # Unless specified by the test, choose a non-conflicting database name, + # to avoid conflicts when running against existing server. + dbname = t.get('dbname', + fallback_dbname.format(test_dir['name'])) + + test_command_base = [ runner.full_path(), '--inputdir', t.get('inputdir', test_dir['sd']), '--expecteddir', t.get('expecteddir', test_dir['sd']), - '--outputdir', test_output, - '--temp-instance', test_output / 'tmp_check', '--bindir', '', '--dlpath', test_dir['bd'], '--max-concurrent-tests=20', - '--port', testport.to_string(), + '--dbname', dbname, ] + t.get('regress_args', []) + test_selection = [] if t.has_key('schedule') - test_command += ['--schedule', t['schedule'],] + test_selection += ['--schedule', t['schedule'],] endif if kind == 'isolation' - test_command += t.get('specs', []) + test_selection += t.get('specs', []) else - test_command += t.get('sql', []) + test_selection += t.get('sql', []) endif env = test_env env.prepend('PATH', temp_install_bindir, test_dir['bd']) test_kwargs = { - 'suite': [test_dir['name']], 'priority': 10, 'timeout': 1000, 'depends': test_deps + t.get('deps', []), 'env': env, } + t.get('test_kwargs', {}) - test(test_dir['name'] / kind, + test(test_group / kind, python, - args: testwrap_base + [ + args: [ + testwrap_base, + '--testgroup', test_group, '--testname', kind, - '--', test_command, + '--', + test_command_base, + '--outputdir', test_output, + '--temp-instance', test_output / 'tmp_check', + '--port', testport.to_string(), + test_selection, ], + suite: test_group, kwargs: test_kwargs, ) + install_suites += test_group + + # some tests can't support running against running DB + if runningcheck and t.get('runningcheck', true) + test(test_group_running / kind, + python, + args: [ + testwrap_base, + '--testgroup', test_group_running, + '--testname', kind, + '--', + test_command_base, + '--outputdir', test_output_running, + test_selection, + ], + is_parallel: t.get('runningcheck-parallel', true), + suite: test_group_running, + kwargs: test_kwargs, + ) + running_suites += test_group_running + endif testport += 1 elif kind == 'tap' @@ -3011,9 +3063,10 @@ foreach test_dir : tests env.set(name, value) endforeach + test_group = test_dir['name'] test_kwargs = { 'protocol': 'tap', - 'suite': [test_dir['name']], + 'suite': test_group, 'timeout': 1000, 'depends': test_deps + t.get('deps', []), 'env': env, @@ -3033,12 +3086,14 @@ foreach test_dir : tests python, kwargs: test_kwargs, args: testwrap_base + [ + '--testgroup', test_dir['name'], '--testname', onetap_p, '--', test_command, test_dir['sd'] / onetap, ], ) endforeach + install_suites += test_group else error('unknown kind @0@ of test in @1@'.format(kind, test_dir['sd'])) endif @@ -3047,6 +3102,14 @@ foreach test_dir : tests endforeach # directories with tests +# repeat condition so meson realizes version dependency +if meson.version().version_compare('>=0.57') + add_test_setup('tmp_install', + is_default: true, + exclude_suites: running_suites) + add_test_setup('running', + exclude_suites: ['setup'] + install_suites) +endif ############################################################### diff --git a/contrib/basic_archive/meson.build b/contrib/basic_archive/meson.build index c30dcfa5d41..08d72598e9d 100644 --- a/contrib/basic_archive/meson.build +++ b/contrib/basic_archive/meson.build @@ -25,5 +25,8 @@ tests += { 'regress_args': [ '--temp-config', files('basic_archive.conf'), ], + # Disabled because these tests require "shared_preload_libraries=basic_archive", + # which typical runningcheck users do not have (e.g. buildfarm clients). + 'runningcheck': false, }, } diff --git a/contrib/pg_freespacemap/meson.build b/contrib/pg_freespacemap/meson.build index 904b37b6e9b..f802e297b67 100644 --- a/contrib/pg_freespacemap/meson.build +++ b/contrib/pg_freespacemap/meson.build @@ -33,5 +33,8 @@ tests += { 'regress_args': [ '--temp-config', files('pg_freespacemap.conf') ], + # Disabled because these tests require "autovacuum=off", which + # typical runningcheck users do not have (e.g. buildfarm clients). + 'runningcheck': false, }, } diff --git a/contrib/pg_stat_statements/meson.build b/contrib/pg_stat_statements/meson.build index 854df138e76..bd0e42331eb 100644 --- a/contrib/pg_stat_statements/meson.build +++ b/contrib/pg_stat_statements/meson.build @@ -41,5 +41,9 @@ tests += { 'pg_stat_statements', ], 'regress_args': ['--temp-config', files('pg_stat_statements.conf')], + # Disabled because these tests require + # "shared_preload_libraries=pg_stat_statements", which typical + # runningcheck users do not have (e.g. buildfarm clients). + 'runningcheck': false, }, } diff --git a/contrib/pg_walinspect/meson.build b/contrib/pg_walinspect/meson.build index 4314a3182a2..351f184de83 100644 --- a/contrib/pg_walinspect/meson.build +++ b/contrib/pg_walinspect/meson.build @@ -28,6 +28,9 @@ tests += { 'sql': [ 'pg_walinspect', ], + # Disabled because these tests require "wal_level=replica", which + # some runningcheck users do not have (e.g. buildfarm clients). 'regress_args': ['--temp-config', files('walinspect.conf')], + 'runningcheck': false, }, } diff --git a/contrib/test_decoding/meson.build b/contrib/test_decoding/meson.build index 6876792fd86..65dfd3f7d88 100644 --- a/contrib/test_decoding/meson.build +++ b/contrib/test_decoding/meson.build @@ -43,6 +43,9 @@ tests += { 'regress_args': [ '--temp-config', files('logical.conf'), ], + # Disabled because these tests require "wal_level=logical", which + # typical runningcheck users do not have (e.g. buildfarm clients). + 'runningcheck': false, }, 'isolation': { 'specs': [ @@ -61,6 +64,8 @@ tests += { 'regress_args': [ '--temp-config', files('logical.conf'), ], + # see above + 'runningcheck': false, }, 'tap': { 'tests': [ diff --git a/src/interfaces/ecpg/test/meson.build b/src/interfaces/ecpg/test/meson.build index 94b26d10314..d4f1f1c0fcf 100644 --- a/src/interfaces/ecpg/test/meson.build +++ b/src/interfaces/ecpg/test/meson.build @@ -84,6 +84,7 @@ tests += { 'test_kwargs': { 'depends': ecpg_test_dependencies, }, + 'dbname': 'ecpg1_regression,ecpg2_regression', 'regress_args': ecpg_regress_args, }, } diff --git a/doc/src/sgml/installation.sgml b/doc/src/sgml/installation.sgml index 9c7f019392e..f64f3290447 100644 --- a/doc/src/sgml/installation.sgml +++ b/doc/src/sgml/installation.sgml @@ -2108,6 +2108,12 @@ ninja detailed information about interpreting the test results. You can repeat this test at any later time by issuing the same command. </para> + + <para> + To run pg_regress and pg_isolation_regress tests against a running + postgres instance, specify <userinput>--setup running</userinput> as an + argument to <userinput>meson test</userinput>. + </para> </step> <step id="meson-install"> diff --git a/.cirrus.yml b/.cirrus.yml index f31923333ef..0113799a6e2 100644 --- a/.cirrus.yml +++ b/.cirrus.yml @@ -195,6 +195,21 @@ task: meson test $MTEST_ARGS --num-processes ${TEST_JOBS} EOF + # test runningcheck, freebsd chosen because it's currently fast enough + test_running_script: | + su postgres <<-EOF + set -e + ulimit -c unlimited + meson test $MTEST_ARGS --quiet --suite setup + export LD_LIBRARY_PATH="$(pwd)/build/tmp_install/usr/local/pgsql/lib/:$LD_LIBRARY_PATH" + mkdir -p build/testrun + build/tmp_install/usr/local/pgsql/bin/initdb -N build/runningcheck --no-instructions -A trust + echo "include '$(pwd)/src/tools/ci/pg_ci_base.conf'" >> build/runningcheck/postgresql.conf + build/tmp_install/usr/local/pgsql/bin/pg_ctl -c -o '-c fsync=off' -D build/runningcheck -l build/testrun/runningcheck.log start + meson test $MTEST_ARGS --num-processes ${TEST_JOBS} --setup running + build/tmp_install/usr/local/pgsql/bin/pg_ctl -D build/runningcheck stop + EOF + on_failure: <<: *on_failure_meson cores_script: src/tools/ci/cores_backtrace.sh freebsd /tmp/cores diff --git a/src/test/isolation/meson.build b/src/test/isolation/meson.build index e2ed400a80a..a999791876a 100644 --- a/src/test/isolation/meson.build +++ b/src/test/isolation/meson.build @@ -67,5 +67,6 @@ tests += { 'priority': 40, 'timeout': 1000, }, + 'dbname': 'isolation_regression', }, } diff --git a/src/test/modules/commit_ts/meson.build b/src/test/modules/commit_ts/meson.build index 60cb12164d2..fa86e70e880 100644 --- a/src/test/modules/commit_ts/meson.build +++ b/src/test/modules/commit_ts/meson.build @@ -6,6 +6,9 @@ tests += { 'sql': [ 'commit_timestamp', ], + # Disabled because these tests require "track_commit_timestamp = on", + # which typical runningcheck users do not have (e.g. buildfarm clients). + 'runningcheck': false, }, 'tap': { 'tests': [ diff --git a/src/test/modules/snapshot_too_old/meson.build b/src/test/modules/snapshot_too_old/meson.build index efd3f1f113b..5094b52949b 100644 --- a/src/test/modules/snapshot_too_old/meson.build +++ b/src/test/modules/snapshot_too_old/meson.build @@ -10,5 +10,8 @@ tests += { 'sto_using_hash_index', ], 'regress_args': ['--temp-config', files('sto.conf')], + # Disabled because these tests require "old_snapshot_threshold" >= 0, which + # typical runningcheck users do not have (e.g. buildfarm clients). + 'runningcheck': false, }, } diff --git a/src/test/modules/test_oat_hooks/meson.build b/src/test/modules/test_oat_hooks/meson.build index 8802bbbac55..e9d09d92c30 100644 --- a/src/test/modules/test_oat_hooks/meson.build +++ b/src/test/modules/test_oat_hooks/meson.build @@ -25,5 +25,6 @@ tests += { 'test_oat_hooks', ], 'regress_args': ['--no-locale', '--encoding=UTF8'], + 'runningcheck': false, }, } diff --git a/src/test/modules/test_pg_dump/meson.build b/src/test/modules/test_pg_dump/meson.build index 41021829f3a..73f1fcf9428 100644 --- a/src/test/modules/test_pg_dump/meson.build +++ b/src/test/modules/test_pg_dump/meson.build @@ -13,6 +13,8 @@ tests += { 'sql': [ 'test_pg_dump', ], + # doesn't delete its user + 'runningcheck': false, }, 'tap': { 'tests': [ diff --git a/src/test/modules/test_slru/meson.build b/src/test/modules/test_slru/meson.build index ca4633c793e..74dacd11ac8 100644 --- a/src/test/modules/test_slru/meson.build +++ b/src/test/modules/test_slru/meson.build @@ -31,5 +31,6 @@ tests += { 'test_slru', ], 'regress_args': ['--temp-config', files('test_slru.conf')], + 'runningcheck': false, }, } diff --git a/src/test/modules/worker_spi/meson.build b/src/test/modules/worker_spi/meson.build index a4a158c75b9..660d721eea9 100644 --- a/src/test/modules/worker_spi/meson.build +++ b/src/test/modules/worker_spi/meson.build @@ -30,6 +30,8 @@ tests += { 'sql': [ 'worker_spi', ], - 'regress_args': ['--temp-config', files('dynamic.conf'), '--dbname=contrib_regression'], + 'dbname': 'contrib_regression', + 'regress_args': ['--temp-config', files('dynamic.conf')], + 'runningcheck': false, }, } diff --git a/src/test/regress/meson.build b/src/test/regress/meson.build index 72a23737fa7..fbca624948b 100644 --- a/src/test/regress/meson.build +++ b/src/test/regress/meson.build @@ -75,5 +75,6 @@ tests += { 'priority': 50, 'timeout': 1000, }, + 'dbname': 'regression', }, } -- 2.38.0
