On Sun, Jul 09 2023, David Bremner wrote:
Note: I almost commented cases where it was not ensured that variable
is defined, to notice later that those were actually defined for sure
-- so in some previous mails in this series I may have missed
same cases.
The comments for this message may be more accurate ;D
Tomi
> We put some effort into testing the built copy rather than some
> installed copy. On the other hand for people like packagers, testing
> the installed copy is also of interest.
>
> When NOTMUCH_TEST_INSTALLED is set to a nonempty value, tests do not
> require a built notmuch tree or running configure.
>
> Some of the tests marked as broken when running against installed
> notmuch are probably fixable.
> ---
> test/README| 17 +
> test/T000-basic.sh | 1 +
> test/T010-help-test.sh | 3 +++
> test/T160-json.sh | 2 ++
> test/T240-dump-restore.sh | 2 ++
> test/T310-emacs.sh | 2 ++
> test/T350-crypto.sh| 1 +
> test/T355-smime.sh | 2 ++
> test/T360-symbol-hiding.sh | 4
> test/T390-python.sh| 4
> test/T391-python-cffi.sh | 4
> test/T395-ruby.sh | 8 ++--
> test/T410-argument-parsing.sh | 4
> test/T480-hex-escaping.sh | 4
> test/T490-parse-time-string.sh | 4
> test/T550-db-features.sh | 4
> test/T566-lib-message.sh | 4
> test/T592-thread-breakage.sh | 4
> test/T710-message-id.sh| 4
> test/T800-asan.sh | 5 +
> test/T850-git.sh | 6 ++
> test/export-dirs.sh| 2 +-
> test/notmuch-test | 12 ++--
> test/test-lib-common.sh| 13 ++---
> test/test-lib-emacs.sh | 12 ++--
> test/test-lib.sh | 27 +--
> 26 files changed, 139 insertions(+), 16 deletions(-)
>
> diff --git a/test/README b/test/README
> index 10f127cb..a81808b1 100644
> --- a/test/README
> +++ b/test/README
> @@ -137,6 +137,23 @@ detection of missing prerequisites. In the future we may
> treat tests
> unable to run because of missing prerequisites, but not explicitly
> skipped by the user, as failures.
>
> +Testing installed notmuch
> +-
> +
> +Systems integrators (e.g. Linux distros) may wish to test an installed
> +version of notmuch. This can be done be running
> +
> + $ NOTMUCH_TEST_INSTALLED=1 ./test/notmuch-test
> +
> +In this scenario the test suite does not assume a built tree, and in
> +particular cannot rely on the output of 'configure'. You may want to
> +set certain feature environment variables ('NOTMUCH_HAVE_*') directly
> +if you know those apply to your installed notmuch). Consider also
> +setting TERM=dumb if the value of TERM cannot be used (e.g. in a
> +chroot with missing terminfo). Note that having a built tree may cause
> +surprising/broken results for NOTMUCH_TEST_INSTALLED, so consider
> +cleaning first.
> +
> Writing Tests
> -
> The test script is written as a shell script. It is to be named as
> diff --git a/test/T000-basic.sh b/test/T000-basic.sh
> index a2f4d93f..642f918d 100755
> --- a/test/T000-basic.sh
> +++ b/test/T000-basic.sh
> @@ -66,6 +66,7 @@ test_begin_subtest 'NOTMUCH_CONFIG is set and points to an
> existing file'
> test_expect_success 'test -f "${NOTMUCH_CONFIG}"'
>
> test_begin_subtest 'PATH is set to build directory'
> +test_subtest_broken_for_installed
> test_expect_equal \
> "$(dirname ${TEST_DIRECTORY})" \
> "$(echo $PATH|cut -f1 -d: | sed -e 's,/test/valgrind/bin$,,')"
> diff --git a/test/T010-help-test.sh b/test/T010-help-test.sh
> index 1fbc46a2..827edc14 100755
> --- a/test/T010-help-test.sh
> +++ b/test/T010-help-test.sh
> @@ -19,6 +19,9 @@ if [ "${NOTMUCH_HAVE_MAN-0}" = "1" ]; then
> test_begin_subtest 'notmuch help tag'
> test_expect_success 'notmuch help tag'
> else
> +if [ -n "${NOTMUCH_TEST_INSTALLED-}" ]; then
> + test_done
> +fi
> test_begin_subtest 'notmuch --help tag (man pages not available)'
> test_expect_success 'test_must_fail notmuch --help tag >/dev/null'
>
> diff --git a/test/T160-json.sh b/test/T160-json.sh
> index f61c3f2d..318c9788 100755
> --- a/test/T160-json.sh
> +++ b/test/T160-json.sh
> @@ -65,6 +65,7 @@ test_expect_equal_json "$output" "[{\"thread\": \"XXX\",
> \"tags\": [\"inbox\",
> \"unread\"]}]"
>
> +if [ -z "${NOTMUCH_TEST_INSTALLED-}" ]; then
> test_begin_subtest "Search message: json, 64-bit timestamp"
> if [ "${NOTMUCH_HAVE_64BIT_TIME_T-0}" != "1" ]; then
> test_subtest_known_broken
> @@ -81,6 +82,7 @@ test_expect_equal_json "$output" "[{\"thread\": \"XXX\",
> \"query\": [\"id:$gen_msg_id\", null],
> \"tags\": [\"inbox\",
> \"unread\"]}]"
> +fi # NOTMUCH_TEST_INSTALLED undefined / empty
>
> test_begin_subtest "Format version: too low"
> test_expe
