Re: Test suites
Neil Williams wrote: On Sun, 2008-04-20 at 14:58 +0200, Martin Fuzzey wrote: Hi mentors, A little question regarding automatic test suites : When a package provides such a suite should the normal package build process : 1) Always run the test suite (for example to catch bugs that may not occur on the developper's architecture) Yes. (That is the main point of having a test suite.) I agree with this (of course with the caveat that DEB_BUILD_OPTIONS=nocheck should disable the test suite). But I'll add a minor and maybe obvious exception -- one should disable or rework any parts of the test suite that would need any of the following: * An X server (e.g. for GUI tests) * A working network connection * Write access to the file system outside the build directory (with obvious exceptions like /tmp) * Interactivity None of these are guaranteed to be available on the buildds. (Although minimal interactivity could be scripted with expect or some such.) best regards, -- Kevin B. McCarty [EMAIL PROTECTED] WWW: http://www.starplot.org/ WWW: http://people.debian.org/~kmccarty/ GPG: public key ID 4F83C751 signature.asc Description: OpenPGP digital signature
Re: Test suites
On Mon, 2008-04-21 at 10:14 -0700, Kevin B. McCarty wrote: Neil Williams wrote: 1) Always run the test suite (for example to catch bugs that may not occur on the developper's architecture) Yes. (That is the main point of having a test suite.) I agree with this (of course with the caveat that DEB_BUILD_OPTIONS=nocheck should disable the test suite). But I'll add a minor and maybe obvious exception -- one should disable or rework any parts of the test suite that would need any of the following: * An X server (e.g. for GUI tests) * A working network connection * Write access to the file system outside the build directory (with obvious exceptions like /tmp) * Interactivity IMHO test suites that require any of these external requirements are simply not sane. I would therefore like to qualify the original statement as you describe, Kevin - a sane test suite should always be run and a test suite that is not sane should be fixed upstream. Wherever possible, the remainder of the test suite should be enabled. -- Neil Williams [EMAIL PROTECTED] signature.asc Description: This is a digitally signed message part
Re: Test suites
Neil Williams [EMAIL PROTECTED] writes: On Mon, 2008-04-21 at 10:14 -0700, Kevin B. McCarty wrote: Neil Williams wrote: 1) Always run the test suite (for example to catch bugs that may not occur on the developper's architecture) Yes. (That is the main point of having a test suite.) I agree with this (of course with the caveat that DEB_BUILD_OPTIONS=nocheck should disable the test suite). But I'll add a minor and maybe obvious exception -- one should disable or rework any parts of the test suite that would need any of the following: * An X server (e.g. for GUI tests) * A working network connection * Write access to the file system outside the build directory (with obvious exceptions like /tmp) * Interactivity IMHO test suites that require any of these external requirements are simply not sane. Agreed, if we're talking specifically about a *unit test* suite, as opposed to the other kinds of tests that should also be in a test suite: functional tests, stress tests, etc. Unit tests should go to whatever lengths are necessary to test *only* the tiny portion of the code that is the subject of the specific test case, and use test doubles for any external dependencies as listed above. With that description of unit tests, it then makes sense to say that the build daemon should always run the unit test suite. Therefore it should also be easy to *only* run the unit tests, without any other tests. -- \ Instead of a trap door, what about a trap window? The guy | `\ looks out it, and if he leans too far, he falls out. Wait. I | _o__) guess that's like a regular window. -- Jack Handey | Ben Finney -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Test suites
Hi mentors, A little question regarding automatic test suites : When a package provides such a suite should the normal package build process : 1) Always run the test suite (for example to catch bugs that may not occur on the developper's architecture) 2) Never run the test suite (to avoid wasting buildd time) 3) Run the test suite provided its resource usage is reasonable (what is reasonable??) As a related question: If the test suite requires extra packages that would not otherwise be needed but the test suite is NOT run as part of the build process should they be listed in Build-Depends anyway? If not how should the requirements be documented? (maybe there should be a field Build-Suggests...) Thanks, Martin -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Test suites
On Sun, 2008-04-20 at 14:58 +0200, Martin Fuzzey wrote: Hi mentors, A little question regarding automatic test suites : When a package provides such a suite should the normal package build process : 1) Always run the test suite (for example to catch bugs that may not occur on the developper's architecture) Yes. (That is the main point of having a test suite.) 2) Never run the test suite (to avoid wasting buildd time) No. 3) Run the test suite provided its resource usage is reasonable (what is reasonable??) A broken test suite should be fixed, not disabled. As a related question: If the test suite requires extra packages that would not otherwise be needed but the test suite is NOT run as part of the build process should they be listed in Build-Depends anyway? If not how should the requirements be documented? (maybe there should be a field Build-Suggests...) The test suite should be run by the buildd so the dependencies of the test suite are the build-dependencies of the source package. Always run the test suite - *except* if the nocheck option is passed to DEB_BUILD_OPTIONS so that it can be easily disabled when cross building. ifeq (,$(findstring nocheck,$(DEB_BUILD_OPTIONS))) make check endif http://wiki.debian.org/EmdebianGuide -- Neil Williams = http://www.data-freedom.org/ http://www.nosoftwarepatents.com/ http://www.linux.codehelp.co.uk/ signature.asc Description: This is a digitally signed message part
Re: Test suites
Neil Williams wrote: make check Please use $(MAKE) instead of calling make directly. Cheers, -- Atomo64 - Raphael Please avoid sending me Word, PowerPoint or Excel attachments. See http://www.gnu.org/philosophy/no-word-attachments.html -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Test suites
On Sun, 2008-04-20 at 13:06 -0500, Raphael Geissert wrote: Neil Williams wrote: make check Please use $(MAKE) instead of calling make directly. Good point - updated the Wiki with the same. -- Neil Williams [EMAIL PROTECTED] signature.asc Description: This is a digitally signed message part
