[PATCH 2/2] test: Have notmuch-test use the modular test collections

2010-05-05 Thread Jesse Rosenthal
These two patches, it turns out, do not quite go all the way -- they do
not set up the environment to be exactly as it was in the full test
suite. (I discovered this because my modular test suite did not fail on
the recent xapian error -- failure being success in this case.)

Further patches, which will be sent tomorrow, will take care of this in
a very naive way: essentially repeating all the steps from the original
suite up to each set of tests -- this means, of course, that the full
test suite will take longer to run, since it will have to recreate the
nm database for each collection. It's still quick enough though, and
this can be improved if we change the expected results for each tests to
conform to more uniquely tailored setups, instead of the leftovers from
previous tests.

Best,
Jesse




Re: [PATCH 2/2] test: Have notmuch-test use the modular test collections

2010-05-04 Thread Jesse Rosenthal
These two patches, it turns out, do not quite go all the way -- they do
not set up the environment to be exactly as it was in the full test
suite. (I discovered this because my modular test suite did not fail on
the recent xapian error -- failure being success in this case.)

Further patches, which will be sent tomorrow, will take care of this in
a very naive way: essentially repeating all the steps from the original
suite up to each set of tests -- this means, of course, that the full
test suite will take longer to run, since it will have to recreate the
nm database for each collection. It's still quick enough though, and
this can be improved if we change the expected results for each tests to
conform to more uniquely tailored setups, instead of the leftovers from
previous tests.

Best,
Jesse


___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH 2/2] test: Have notmuch-test use the modular test collections

2010-04-30 Thread Jesse Rosenthal
Introduce two new functions (run_collection_setup_and_tests and
run_test_suite) to notmuch-test. The first takes as an argument a subdir
of test/tests and runs all .setup and .test files in there in order of
their numerical prefix. The latter either runs all tests (if given "all"
or no argument) or a given dir (if given it as an argument).

Also introduce the variable TEST_COLLECTIONS. These are all the folders
that "notmuch-test all" should look for. It won't automatically run all
folders, in case there is something that is only relevant in certain
circumstances (such as, perhaps, future tests for remote usage).
---
 test/notmuch-test |  921 +++--
 1 files changed, 47 insertions(+), 874 deletions(-)

diff --git a/test/notmuch-test b/test/notmuch-test
index a861df1..29d5c91 100755
--- a/test/notmuch-test
+++ b/test/notmuch-test
@@ -7,6 +7,20 @@ set -e
 # without regard to the time zone of where the test suite is run.
 export TZ=UTC+8

+TEST_COLLECTION_DIR="$(dirname $(readlink -f $0))/tests"
+
+TEST_COLLECTIONS="notmuch-new
+notmuch-search
+json
+thread-naming
+notmuch-reply
+uuencoded
+notmuch-dump-restore
+out-of-order-threading
+author-reordering
+from-guessing-multiple
+from-guessing-single"
+
 find_notmuch_binary ()
 {
 dir=$1
@@ -210,13 +224,15 @@ notmuch_show_sanitize ()
 sed -e "$NOTMUCH_SHOW_FILENAME_SQUELCH"
 }

-rm -rf ${TEST_DIR}
-mkdir ${TEST_DIR}
-cd ${TEST_DIR}
+initialize_notmuch_test ()
+{
+rm -rf ${TEST_DIR}
+mkdir ${TEST_DIR}
+cd ${TEST_DIR}

-mkdir ${MAIL_DIR}
+mkdir ${MAIL_DIR}

-cat < ${NOTMUCH_CONFIG}
+cat < ${NOTMUCH_CONFIG}
 [database]
 path=${MAIL_DIR}

@@ -225,878 +241,35 @@ name=Notmuch Test Suite
 primary_email=test_suite at notmuchmail.org
 other_email=test_suite_other at notmuchmail.org;test_suite at otherdomain.org
 EOF
+}

-printf "Testing \"notmuch new\" in several variations:\n"
-printf " No new messages...\t\t\t\t"
-output=$(NOTMUCH_NEW)
-pass_if_equal "$output" "No new mail."
-
-printf " Single new message...\t\t\t\t"
-generate_message
-output=$(NOTMUCH_NEW)
-pass_if_equal "$output" "Added 1 new message to the database."
-
-printf " Multiple new messages...\t\t\t"
-generate_message
-generate_message
-output=$(NOTMUCH_NEW)
-pass_if_equal "$output" "Added 2 new messages to the database."
-
-printf " No new messages (non-empty DB)...\t\t"
-output=$(NOTMUCH_NEW)
-pass_if_equal "$output" "No new mail."
-
-printf " New directories...\t\t\t\t"
-rm -rf ${MAIL_DIR}/* ${MAIL_DIR}/.notmuch
-mkdir ${MAIL_DIR}/def
-mkdir ${MAIL_DIR}/ghi
-generate_message [dir]=def
-
-output=$(NOTMUCH_NEW)
-pass_if_equal "$output" "Added 1 new message to the database."
-
-printf " Alternate inode order...\t\t\t"
-
-rm -rf ${MAIL_DIR}/.notmuch
-mv ${MAIL_DIR}/ghi ${MAIL_DIR}/abc
-rm ${MAIL_DIR}/def/*
-generate_message [dir]=abc
-
-output=$(NOTMUCH_NEW)
-pass_if_equal "$output" "Added 1 new message to the database."
-
-printf " Message moved in...\t\t\t\t"
-rm -rf ${MAIL_DIR}/* ${MAIL_DIR}/.notmuch
-generate_message
-tmp_msg_filename=tmp/$gen_msg_filename
-mkdir -p $(dirname $tmp_msg_filename)
-mv $gen_msg_filename $tmp_msg_filename
-increment_mtime ${MAIL_DIR}
-$NOTMUCH new > /dev/null
-mv $tmp_msg_filename $gen_msg_filename
-increment_mtime ${MAIL_DIR}
-output=$(NOTMUCH_NEW)
-pass_if_equal "$output" "Added 1 new message to the database."
-
-printf " Renamed message...\t\t\t\t"
-
-generate_message
-$NOTMUCH new > /dev/null
-mv $gen_msg_filename ${gen_msg_filename}-renamed
-increment_mtime ${MAIL_DIR}
-output=$(NOTMUCH_NEW)
-pass_if_equal "$output" "No new mail. Detected 1 file rename."
-
-printf " Deleted message...\t\t\t\t"
-
-rm ${gen_msg_filename}-renamed
-increment_mtime ${MAIL_DIR}
-output=$(NOTMUCH_NEW)
-pass_if_equal "$output" "No new mail. Removed 1 message."
-
-printf " Renamed directory...\t\t\t\t"
-
-generate_message [dir]=dir
-generate_message [dir]=dir
-generate_message [dir]=dir
-
-$NOTMUCH new > /dev/null
-
-mv ${MAIL_DIR}/dir ${MAIL_DIR}/dir-renamed
-increment_mtime ${MAIL_DIR}
-
-output=$(NOTMUCH_NEW)
-pass_if_equal "$output" "No new mail. Detected 3 file renames."
-
-printf " Deleted directory...\t\t\t\t"
-
-rm -rf ${MAIL_DIR}/dir-renamed
-increment_mtime ${MAIL_DIR}
-
-output=$(NOTMUCH_NEW)
-pass_if_equal "$output" "No new mail. Removed 3 messages."
-
-printf " New directory (at end of list)...\t\t"
-
-generate_message [dir]=zzz
-generate_message [dir]=zzz
-generate_message [dir]=zzz
-
-output=$(NOTMUCH_NEW)
-pass_if_equal "$output" "Added 3 new messages to the database."
-
-printf " Deleted directory (end of list)...\t\t"
-
-rm -rf ${MAIL_DIR}/zzz
-increment_mtime ${MAIL_DIR}
-
-output=$(NOTMUCH_NEW)
-pass_if_equal "$output" "No new mail. Removed 3 messages."
-
-printf " New symlink to directory...\t\t\t"
-
-rm -rf ${MAIL_DIR}/.notmuch
-mv ${MAIL_DIR} ${TEST_DIR}/actual_maildir
-
-mkdir ${MAIL_DIR}
-ln -s ${TEST_DIR}/actual_maildir ${MAIL_DIR}/symlink
-
-output=$(NOTMUCH_NEW)
-pass_if_equal "$output"