Re: pytest integration for the notmuch test suite
On Sun, Mar 25 2018, David Bremner wrote:
> Here's one approach. A given pytest "file" can be embedded in a normal
> (for us) test script. As I write this, it occurs to me you might be
> thinking of embedding unit tests in the bindings source files; that
> would be easy to add, something along the lines of
>
> test_begin_subtest "python bindings embedded unit tests"
> test_expect_success "${NOTMUCH_PYTEST}
> ${NOTMUCH_SRCDIR}/bindings/python/notmuch"
Hmm.
Looks a bit strange to embed the pytest snippets into shell script and then
execute each of these individiually. The only thing py.test seems to do here is
"visualizing" assert output. We could just use normal python otherwise, and
just not (necessarily) drop things into functions.
If we had pytest, I'd suggest the tests were written and executed
separately (from one test script) and then results collected somehow
to the final aggregator.
(Recently I've been working with py.test-3 (and am not too happy with it,
perhaps it is better when testing python code, we use it for testing c code
with help of ctypes...) so I have some knowledge of how it works...
BTW: in case of pytest. be careful there is no `conftest.py` in any of the
directories starting / !!!, to mess up with your tests >;/
> You could also run one source file of tests with
>
> test_begin_subtest "python bindings foo tests"
> test_expect_success "${NOTMUCH_PYTEST}
> ${NOTMUCH_SRCDIR}/bindings/python/notmuch/test_foo.py"
notmuch bätter... ;/
>
> that would give a less granular result, at the cost of more boilerplate
Tomi
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch
[PATCH 2/3] test: add new test_expect_pytest_success
As the name suggests, this is something of a cross between
test_python (reading a script from stdin) and test expect success.
It seemed somewhat redundant to allow our usual kind of file
comparison with pytest scripts, although that will make it tougher to
compare output with the CLI.
---
test/test-lib.sh | 7 +++
1 file changed, 7 insertions(+)
diff --git a/test/test-lib.sh b/test/test-lib.sh
index 5b212514..fd0e9647 100644
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -1002,6 +1002,13 @@ test_python() {
$NOTMUCH_PYTHON -B - > OUTPUT
}
+test_expect_pytest_success() {
+test_file="test_${test_count}.py"
+cat > ${test_file}
+PYTHONPATH="$NOTMUCH_SRCDIR/bindings/python${PYTHONPATH:+:$PYTHONPATH}" \
+ test_expect_success "$NOTMUCH_PYTEST ${test_file}"
+}
+
test_ruby() {
MAIL_DIR=$MAIL_DIR ruby -I $NOTMUCH_SRCDIR/bindings/ruby> OUTPUT
}
--
2.16.2
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch
[PATCH 1/3] configure: check for pytest binary
This is to support future use of pytest in the test suite
---
configure | 24
1 file changed, 24 insertions(+)
diff --git a/configure b/configure
index b177b141..ab45878d 100755
--- a/configure
+++ b/configure
@@ -62,6 +62,7 @@ CXXFLAGS=${CXXFLAGS:-\$(CFLAGS)}
LDFLAGS=${LDFLAGS:-}
XAPIAN_CONFIG=${XAPIAN_CONFIG:-}
PYTHON=${PYTHON:-}
+PYTEST=${PYTEST:-}
# We don't allow the EMACS or GZIP Makefile variables inherit values
# from the environment as we do with CC and CXX above. The reason is
@@ -118,6 +119,8 @@ Other environment variables can be used to control
configure itself,
library. [$XAPIAN_CONFIG]
PYTHON Name of python command to use in
configure and the test suite.
+PYTEST Name of pytest command to use in
+the test suite.
Additionally, various options can be specified on the configure
command line.
@@ -571,6 +574,24 @@ if [ $have_python -eq 0 ]; then
errors=$((errors + 1))
fi
+pytest=""
+if [ $have_python -eq 1 ]; then
+printf "Checking for pytest... "
+have_pytest=0
+
+for name in ${PYTEST} pytest-3 pytest pytest-2; do
+if command -v $name > /dev/null; then
+have_pytest=1
+pytest=$name
+printf "Yes (%s).\n" $pytest
+break
+fi
+done
+if [ $have_pytest -eq 0 ]; then
+printf "No (some tests may be skipped).\n"
+fi
+fi
+
printf "Checking for valgrind development files... "
if pkg-config --exists valgrind; then
printf "Yes.\n"
@@ -1234,6 +1255,9 @@ NOTMUCH_HAVE_MAN=$((have_sphinx))
# Name of python interpreter
NOTMUCH_PYTHON=${python}
+# Name of pytest runner
+NOTMUCH_PYTEST=${pytest}
+
# Are the ruby development files (and ruby) available? If not skip
# building/testing ruby bindings.
NOTMUCH_HAVE_RUBY_DEV=${have_ruby_dev}
--
2.16.2
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch
pytest integration for the notmuch test suite
Here's one approach. A given pytest "file" can be embedded in a normal
(for us) test script. As I write this, it occurs to me you might be
thinking of embedding unit tests in the bindings source files; that
would be easy to add, something along the lines of
test_begin_subtest "python bindings embedded unit tests"
test_expect_success "${NOTMUCH_PYTEST}
${NOTMUCH_SRCDIR}/bindings/python/notmuch"
You could also run one source file of tests with
test_begin_subtest "python bindings foo tests"
test_expect_success "${NOTMUCH_PYTEST}
${NOTMUCH_SRCDIR}/bindings/python/notmuch/test_foo.py"
that would give a less granular result, at the cost of more boilerplate
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch
[PATCH 3/3] test: add example test using pytest
It might make sense to remove the non-pytest version of this test, but
that requires other changes to following tests
---
test/T390-python.sh | 12
1 file changed, 12 insertions(+)
diff --git a/test/T390-python.sh b/test/T390-python.sh
index 9f71ce3c..72036004 100755
--- a/test/T390-python.sh
+++ b/test/T390-python.sh
@@ -194,4 +194,16 @@ EOF
echo "$fname" > EXPECTED
test_expect_equal_file EXPECTED OUTPUT
+if [ -n "${NOTMUCH_PYTEST}" ]; then
+
+test_begin_subtest "pytest based get_revision"
+test_expect_pytest_success
