Re: pytest integration for the notmuch test suite
Floris Bruynooghewrites: > On Sun, Mar 25 2018, David Bremner wrote: > >> Here's one approach. A given pytest "file" can be embedded in a normal >> (for us) test script. As I write this, it occurs to me you might be >> thinking of embedding unit tests in the bindings source files; that >> would be easy to add, something along the lines of >> >> test_begin_subtest "python bindings embedded unit tests" >> test_expect_success "${NOTMUCH_PYTEST} >> ${NOTMUCH_SRCDIR}/bindings/python/notmuch" > > I was trying to construct something where a full pytest run on one > python version was one subtest. For granularity I think treating an > entire pytest run as a subtest with just checking the return code should > be sufficient, > e.g. `python2.7 -m pytest ${NOTMUCH_SRCDIR}/bindings/python/notmuch`. > > But the whole test in this case would be this same subtest but once with > python2.7, python3.5, python3.6 etc. > > What do you think of this? The notmuch build system currently looks for a single version of python. I'm pretty sure we don't want to require multiple python versions to build notmuch, although I guess it wouldn't be that hard to detect whatever versions exist in the environment and run pytest with each of them. I just need a list of names to look for (or a better way to find all the pythons installed). Debian has the handy comand "py3versions --installed", but I suppose that's completely unportable. Then each one can be checked to see if it finds the pytest module. It sounds doable to me; it needs a bit more work in configure, but I can drop the stuff in test-lib.sh d ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: pytest integration for the notmuch test suite
On Sun, Mar 25 2018, David Bremner wrote: > Here's one approach. A given pytest "file" can be embedded in a normal > (for us) test script. As I write this, it occurs to me you might be > thinking of embedding unit tests in the bindings source files; that > would be easy to add, something along the lines of > > test_begin_subtest "python bindings embedded unit tests" > test_expect_success "${NOTMUCH_PYTEST} > ${NOTMUCH_SRCDIR}/bindings/python/notmuch" I was trying to construct something where a full pytest run on one python version was one subtest. For granularity I think treating an entire pytest run as a subtest with just checking the return code should be sufficient, e.g. `python2.7 -m pytest ${NOTMUCH_SRCDIR}/bindings/python/notmuch`. But the whole test in this case would be this same subtest but once with python2.7, python3.5, python3.6 etc. What do you think of this? Cheers, Floris ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: pytest integration for the notmuch test suite
Tomi Ollilawrites: > On Sun, Mar 25 2018, David Bremner wrote: > >> Here's one approach. A given pytest "file" can be embedded in a normal >> (for us) test script. As I write this, it occurs to me you might be >> thinking of embedding unit tests in the bindings source files; that >> would be easy to add, something along the lines of >> >> test_begin_subtest "python bindings embedded unit tests" >> test_expect_success "${NOTMUCH_PYTEST} >> ${NOTMUCH_SRCDIR}/bindings/python/notmuch" > > Hmm. > > Looks a bit strange to embed the pytest snippets into shell script and then > execute each of these individiually. The only thing py.test seems to do here > is > "visualizing" assert output. We could just use normal python otherwise, and > just not (necessarily) drop things into functions. Yes, I had pretty much the same thought. It's also true that pytest allows mixing tests with regular definition, if desired (i.e. it can search for all the functions starting test_ or some other marker). It's also true this doesn't really cost more at run time than the current test_python; they both exec the python interpreter. > If we had pytest, I'd suggest the tests were written and executed > separately (from one test script) and then results collected somehow > to the final aggregator. Sure, that sounds nicer. I just don't want to invest a lot in writing glue before it's clear what is needed. If start by running the pytest files the crude way, we can always improve the glue later. That does argue against doing many embedded snippets, I agree. d ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: pytest integration for the notmuch test suite
On Sun, Mar 25 2018, David Bremner wrote: > Here's one approach. A given pytest "file" can be embedded in a normal > (for us) test script. As I write this, it occurs to me you might be > thinking of embedding unit tests in the bindings source files; that > would be easy to add, something along the lines of > > test_begin_subtest "python bindings embedded unit tests" > test_expect_success "${NOTMUCH_PYTEST} > ${NOTMUCH_SRCDIR}/bindings/python/notmuch" Hmm. Looks a bit strange to embed the pytest snippets into shell script and then execute each of these individiually. The only thing py.test seems to do here is "visualizing" assert output. We could just use normal python otherwise, and just not (necessarily) drop things into functions. If we had pytest, I'd suggest the tests were written and executed separately (from one test script) and then results collected somehow to the final aggregator. (Recently I've been working with py.test-3 (and am not too happy with it, perhaps it is better when testing python code, we use it for testing c code with help of ctypes...) so I have some knowledge of how it works... BTW: in case of pytest. be careful there is no `conftest.py` in any of the directories starting / !!!, to mess up with your tests >;/ > You could also run one source file of tests with > > test_begin_subtest "python bindings foo tests" > test_expect_success "${NOTMUCH_PYTEST} > ${NOTMUCH_SRCDIR}/bindings/python/notmuch/test_foo.py" notmuch bätter... ;/ > > that would give a less granular result, at the cost of more boilerplate Tomi ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch