Re: pytest integration for the notmuch test suite
Floris Bruynooghe writes:
> On Sun, Mar 25 2018, David Bremner wrote:
>
>> Here's one approach. A given pytest "file" can be embedded in a normal
>> (for us) test script. As I write this, it occurs to me you might be
>> thinking of embedding unit tests in the bindings source files; that
>> would be easy to add, something along the lines of
>>
>> test_begin_subtest "python bindings embedded unit tests"
>> test_expect_success "${NOTMUCH_PYTEST}
>> ${NOTMUCH_SRCDIR}/bindings/python/notmuch"
>
> I was trying to construct something where a full pytest run on one
> python version was one subtest. For granularity I think treating an
> entire pytest run as a subtest with just checking the return code should
> be sufficient,
> e.g. `python2.7 -m pytest ${NOTMUCH_SRCDIR}/bindings/python/notmuch`.
>
> But the whole test in this case would be this same subtest but once with
> python2.7, python3.5, python3.6 etc.
>
> What do you think of this?
The notmuch build system currently looks for a single version of
python. I'm pretty sure we don't want to require multiple python
versions to build notmuch, although I guess it wouldn't be that hard to
detect whatever versions exist in the environment and run pytest with
each of them. I just need a list of names to look for (or a better way
to find all the pythons installed). Debian has the handy comand
"py3versions --installed", but I suppose that's completely unportable.
Then each one can be checked to see if it finds the pytest module.
It sounds doable to me; it needs a bit more work in configure, but I can
drop the stuff in test-lib.sh
d
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch
Re: pytest integration for the notmuch test suite
On Sun, Mar 25 2018, David Bremner wrote:
> Here's one approach. A given pytest "file" can be embedded in a normal
> (for us) test script. As I write this, it occurs to me you might be
> thinking of embedding unit tests in the bindings source files; that
> would be easy to add, something along the lines of
>
> test_begin_subtest "python bindings embedded unit tests"
> test_expect_success "${NOTMUCH_PYTEST}
> ${NOTMUCH_SRCDIR}/bindings/python/notmuch"
I was trying to construct something where a full pytest run on one
python version was one subtest. For granularity I think treating an
entire pytest run as a subtest with just checking the return code should
be sufficient,
e.g. `python2.7 -m pytest ${NOTMUCH_SRCDIR}/bindings/python/notmuch`.
But the whole test in this case would be this same subtest but once with
python2.7, python3.5, python3.6 etc.
What do you think of this?
Cheers,
Floris
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH 1/3] configure: check for pytest binary
On Sun, Mar 25 2018, David Bremner wrote:
> This is to support future use of pytest in the test suite
Thanks for having a go at this!
> ---
> configure | 24
> 1 file changed, 24 insertions(+)
>
> diff --git a/configure b/configure
> index b177b141..ab45878d 100755
> --- a/configure
> +++ b/configure
> @@ -62,6 +62,7 @@ CXXFLAGS=${CXXFLAGS:-\$(CFLAGS)}
> LDFLAGS=${LDFLAGS:-}
> XAPIAN_CONFIG=${XAPIAN_CONFIG:-}
> PYTHON=${PYTHON:-}
> +PYTEST=${PYTEST:-}
>
> # We don't allow the EMACS or GZIP Makefile variables inherit values
> # from the environment as we do with CC and CXX above. The reason is
> @@ -118,6 +119,8 @@ Other environment variables can be used to control
> configure itself,
> library. [$XAPIAN_CONFIG]
> PYTHON Name of python command to use in
> configure and the test suite.
> +PYTEST Name of pytest command to use in
> +the test suite.
>
> Additionally, various options can be specified on the configure
> command line.
> @@ -571,6 +574,24 @@ if [ $have_python -eq 0 ]; then
> errors=$((errors + 1))
> fi
>
> +pytest=""
> +if [ $have_python -eq 1 ]; then
> +printf "Checking for pytest... "
> +have_pytest=0
> +
> +for name in ${PYTEST} pytest-3 pytest pytest-2; do
This is kind of not granular enough I think. It would be better to
invoke pytest as "pythonX.Y -m pytest" which is the safe way to execute
it on all python versions.
> +if command -v $name > /dev/null; then
> +have_pytest=1
> +pytest=$name
> +printf "Yes (%s).\n" $pytest
> +break
> +fi
> +done
> +if [ $have_pytest -eq 0 ]; then
> +printf "No (some tests may be skipped).\n"
> +fi
The other thing I was trying to achieve was to be able to run the
unittest for each python version, so say 2.7, 3.5 & 3.6 are supported
then I was trying to find all of those instead of just one.
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch
Re: New Python bindings (was: Crash with Python bindings)
On Wed, Mar 21 2018, Justus Winter wrote: > > Floris Bruynooghe writes: > >> This is exactly what I have fixed in my alternative bindings which I >> created around the end of last year [0]. So we do have an idea of how >> to fix this, at the time I said I do believe that it's possible to also >> do this for the existing bindings even though it is a lot of work. >> After some talking between dkg and me we got to a way forward which >> proposed this, but I must admit that after messing a little with getting >> a pytest run integrated into the notmuch test-suite instead of using tox >> I lost momentum on the project and didn't advance any further. > > I'm sorry that I didn't speak up when you announced your work. I'm > actually excited about a new set of bindings for Python. I agree with > using cffi. I briefly looked at the code, and I believe it is much > nicer than what we currently have. Nice to hear, thanks! > I trust that it works fine with Python 3, does it? The version I made so far *only* works on Python 3. Mostly because it was easier, but it also allows some API niceties. > The testsuite cannot depend on pulling stuff from the net simply because > build servers typically do not have access to it. That is a hard > requirement. Sure I understand that. See another part of this thread on this though. Cheers, Floris ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: pytest integration for the notmuch test suite
Tomi Ollila writes:
> On Sun, Mar 25 2018, David Bremner wrote:
>
>> Here's one approach. A given pytest "file" can be embedded in a normal
>> (for us) test script. As I write this, it occurs to me you might be
>> thinking of embedding unit tests in the bindings source files; that
>> would be easy to add, something along the lines of
>>
>> test_begin_subtest "python bindings embedded unit tests"
>> test_expect_success "${NOTMUCH_PYTEST}
>> ${NOTMUCH_SRCDIR}/bindings/python/notmuch"
>
> Hmm.
>
> Looks a bit strange to embed the pytest snippets into shell script and then
> execute each of these individiually. The only thing py.test seems to do here
> is
> "visualizing" assert output. We could just use normal python otherwise, and
> just not (necessarily) drop things into functions.
Yes, I had pretty much the same thought. It's also true that pytest
allows mixing tests with regular definition, if desired (i.e. it can
search for all the functions starting test_ or some other marker).
It's also true this doesn't really cost more at run time than the
current test_python; they both exec the python interpreter.
> If we had pytest, I'd suggest the tests were written and executed
> separately (from one test script) and then results collected somehow
> to the final aggregator.
Sure, that sounds nicer. I just don't want to invest a lot in writing
glue before it's clear what is needed. If start by running the pytest
files the crude way, we can always improve the glue later. That does
argue against doing many embedded snippets, I agree.
d
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch
