Re: autopkgtest'ing against multiple Python versions
Jakub Wilk writes ("Re: autopkgtest'ing against multiple Python versions"):
> * Yaroslav Halchenko , 2012-04-23, 09:17:
> >>The “failures” was caused by the following misfeature of the
> >>specification: “if a test […] prints to stderr, it is considered to
> >>have failed.” But nosetests does print stuff to stderr even if
> >>everything is all right (see bug #460242).
> >yikes -- doesn't it fail if underlying "test command" returns with
> >non-0 exit code?
>
> Thankfully it does. :)
> s/[…]/exits nonzero, or/ - sorry for the confusion.
I think it's correct to call stderr output a test failure. I wrote
the spec this way because there are far too many programs which don't
deal with exit statuses properly.
If you need to tolerate stderr output, you can redirect it to stdout.
Thanks,
Ian.
--
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive:
http://lists.debian.org/20442.32906.743237.706...@chiark.greenend.org.uk
Re: autopkgtest'ing against multiple Python versions
* Brian Sutherland , 2012-04-30, 19:47: How are you actually running the tests with adt-run. Did you actually setup a testbed, or is there now an easy way to run the tests? When you apply the patch from bug #648148, you can use your host system as testbed. You run this in an unpacked (and built, if needed) source tree: adt-run ./ --- adt-virt-null But since the patch breaks other (unrelated) things, and I found adt-run rather user-unfriendly, and I couldn't easily understand its source code, I ended up writing my own simplistic DEP-8 test runner: https://bitbucket.org/jwilk/debian-misc/src/tip/sadt The script depends on the following packages: python (>= 2.7) | python-argparse, python-debian, devscripts Do you run the tests against the code installed via the .deb package or on the code in the unpacked source? The former. The specification reads: “the tests must test the INSTALLED version of the program.” -- Jakub Wilk -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120430192952.ga4...@jwilk.net
Re: autopkgtest'ing against multiple Python versions
On Sat, Apr 21, 2012 at 10:33:24PM +0200, Jakub Wilk wrote: > 3) Provide one test that is run against all installed versions > (pyversions -i): > > Tests: test-installed > Depends: python-pet-module > > This is a variant of 2 which is more friendly to users who can't or > don't want to install extra packages. It's also less useful for > users who would install test dependencies automatically anyway > (since then you do want to install python-all). This issue could be > fixed if autopkgtest supported weak dependencies, e.g.: > > Tests: test-installed > Depends: python-pet-module > Recommends: python-all This is the way we did it in the zope.* packages. But, as an observer, I'm puzzled by 2 things: * How are you actually running the tests with adt-run. Did you actually setup a testbed, or is there now an easy way to run the tests? * Do you run the tests against the code installed via the .deb package or on the code in the unpacked source? -- Brian Sutherland -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120430174742.ga45...@minimac.lan
Re: autopkgtest'ing against multiple Python versions
* Yaroslav Halchenko , 2012-04-23, 09:17: The “failures” was caused by the following misfeature of the specification: “if a test […] prints to stderr, it is considered to have failed.” But nosetests does print stuff to stderr even if everything is all right (see bug #460242). yikes -- doesn't it fail if underlying "test command" returns with non-0 exit code? Thankfully it does. :) s/[…]/exits nonzero, or/ - sorry for the confusion. IMHO that should be the criterion, not catching stderr, which might have some spurious output from tests which might be testing either application handles erroneous cases correctly (where it might output to stderr). ACK. Even if nosetests printed the usual stuff to stdout, I would still have to redirect stderr to stdout (or /dev/null), because the tested code could rightfully emit warnings. -- Jakub Wilk -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120423143025.ga3...@jwilk.net
Re: autopkgtest'ing against multiple Python versions
On Mon, 23 Apr 2012, Jakub Wilk wrote: > Maybe it was not a good example after all. :) I just realized that, > according to adt-run, the tests were failed. I didn't notice this > before because there was nothing at the end of adt-run's output that > would indicate that something went awry. > The “failures” was caused by the following misfeature of the > specification: “if a test […] prints to stderr, it is considered to > have failed.” But nosetests does print stuff to stderr even if > everything is all right (see bug #460242). yikes -- doesn't it fail if underlying "test command" returns with non-0 exit code? IMHO that should be the criterion, not catching stderr, which might have some spurious output from tests which might be testing either application handles erroneous cases correctly (where it might output to stderr). -- =--= Keep in touch www.onerussian.com Yaroslav Halchenko www.ohloh.net/accounts/yarikoptic -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120423131730.gq9...@onerussian.com
Re: autopkgtest'ing against multiple Python versions
* Jakub Wilk , 2012-04-22, 10:03: would you also mind pointing to the 'best of the breed' package fro which you have created an autopkgtest pkg? http://anonscm.debian.org/viewvc/python-modules/packages/gamera/trunk/debian/tests/ Maybe it was not a good example after all. :) I just realized that, according to adt-run, the tests were failed. I didn't notice this before because there was nothing at the end of adt-run's output that would indicate that something went awry. The “failures” was caused by the following misfeature of the specification: “if a test […] prints to stderr, it is considered to have failed.” But nosetests does print stuff to stderr even if everything is all right (see bug #460242). -- Jakub Wilk -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120423124033.ga8...@jwilk.net
Re: autopkgtest'ing against multiple Python versions
* Yaroslav Halchenko , 2012-04-21, 19:24: 3 (enhanced with Recommends) sounds like a perfect solution to me as well... if only someone hacked it up for autopkgtest? I was tempted even to add Recommends now and wait patiently until it's implemented... but it turns down that I can't. The specification reads "Any unknown fields will cause the whole stanza to be skipped." Ugh. Though, thankfully, it's not really implemented this way: adt-run appears to be ignoring unknown fields. would you also mind pointing to the 'best of the breed' package fro which you have created an autopkgtest pkg? http://anonscm.debian.org/viewvc/python-modules/packages/gamera/trunk/debian/tests/ -- Jakub Wilk -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120422080327.ga5...@jwilk.net
Re: autopkgtest'ing against multiple Python versions
Cool, Jakub, 3 (enhanced with Recommends) sounds like a perfect solution to me as well... if only someone hacked it up for autopkgtest? would you also mind pointing to the 'best of the breed' package fro which you have created an autopkgtest pkg? > (since then you do want to install python-all). This issue could be > fixed if autopkgtest supported weak dependencies, e.g.: > Tests: test-installed > Depends: python-pet-module > Recommends: python-all -- =--= Keep in touch www.onerussian.com Yaroslav Halchenko www.ohloh.net/accounts/yarikoptic -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120421232425.gz9...@onerussian.com
autopkgtest'ing against multiple Python versions
I started adding DEP-8 tests to my packages. It worked reasonably well so far (after I fixed #648148 localy...). However, I have doubts about which strategy to choose when it comes to testing with multiple Python versions. The following schemes come to my mind: 1) Test only with default version: Tests: test-default Depends: python-pet-module, python This is very straight-forward to implement. Also, it looks like that soon we'll have only one supported Python 2.X version anyway, so not testing against non-default versions wouldn't be a big deal. 2) Provide one test that is run against all supported versions (pyversions -r): Tests: test-all Depends: python-pet-module, python-all This is also very easy to implement. The disadvantage is that in some cases a (non-root) user won't be able to run any test even though he has some (or maybe even all) Python interpreters installed. 3) Provide one test that is run against all installed versions (pyversions -i): Tests: test-installed Depends: python-pet-module This is a variant of 2 which is more friendly to users who can't or don't want to install extra packages. It's also less useful for users who would install test dependencies automatically anyway (since then you do want to install python-all). This issue could be fixed if autopkgtest supported weak dependencies, e.g.: Tests: test-installed Depends: python-pet-module Recommends: python-all 4) Provide two tests: - for testing with default version; - for testing with all versions. Tests: test-default Depends: python-pet-module, python Tests: test-all Depends: python-pet-module, python-all It his case one could always run at least the first test. But on the other hand the second one is wasteful... 5) Provide two tests: - for testing with default version; - for testing with non-default versions. Tests: test-default Depends: python-pet-module, python Tests: test-nondefault Depends: python-pet-module, python-all It's a less wasteful variant of 4. However, it's also more difficult to implement. What do you think? I'm currently leaning towards 3. -- Jakub Wilk -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120421203324.ga1...@jwilk.net
