Re: Automated testing - design and interfaces

2005-11-24 Thread Ian Jackson
Anthony Towns writes (Re: Automated testing - design and interfaces): On Mon, Nov 21, 2005 at 06:22:37PM +, Ian Jackson wrote: This is no good because we want the test environment to be able to tell which tests failed, so the test cases have to be enumerated in the test metadata file

Re: Automated testing - design and interfaces

2005-11-23 Thread Anthony Towns
On Mon, Nov 21, 2005 at 06:22:37PM +, Ian Jackson wrote: Note that it's often better to have a single script run many tests, so you probably want to allow tests to pass back some summary information, or include the last ten lines of its output or similar. Something like: foo FAIL:

Re: Automated testing - design and interfaces

2005-11-23 Thread Robert Collins
On Wed, 2005-11-23 at 18:16 +1000, Anthony Towns wrote: On Mon, Nov 21, 2005 at 06:22:37PM +, Ian Jackson wrote: Note that it's often better to have a single script run many tests, so you probably want to allow tests to pass back some summary information, or include the last ten

Re: Automated testing - design and interfaces

2005-11-21 Thread Ian Jackson
Anthony Towns writes (Re: Automated testing - design and interfaces): On Thu, Nov 17, 2005 at 06:43:32PM +, Ian Jackson wrote: The source package provides a test metadata file debian/tests/ control. This is a file containing zero or more RFC822-style stanzas, along these lines

Re: Automated testing - design and interfaces

2005-11-20 Thread Robert Collins
On Thu, 2005-11-17 at 14:36 -0800, Steve Langasek wrote: [let's get this over to a technical list like it was supposed to be ;)] Following your exit status based approach you could add to stanzas something like: Expected-Status: 0 I found the above requirement the very minimum for

Re: Automated testing - design and interfaces

2005-11-18 Thread Stefano Zacchiroli
On Thu, Nov 17, 2005 at 02:36:06PM -0800, Steve Langasek wrote: FWIW, I don't see that there's a clear advantage to having the test harness *expect* non-zero exit values (or non-empty output as you also suggested). As I understood it, the proposed approach is about standardizing and easing the

Re: Automated testing - design and interfaces

2005-11-17 Thread Steve Langasek
[let's get this over to a technical list like it was supposed to be ;)] On Thu, Nov 17, 2005 at 10:43:34PM +0100, Stefano Zacchiroli wrote: On Thu, Nov 17, 2005 at 06:43:32PM +, Ian Jackson wrote: This means execute debian/tests/fred, debian/tests/bill, etc., each with no arguments,

Re: Automated testing - design and interfaces

2005-11-17 Thread Anthony Towns
Bcc'ed to -project; followups to -devel. On Thu, Nov 17, 2005 at 06:43:32PM +, Ian Jackson wrote: Note that the point is to be able to test the _actual package_, _as installed_ (eg on a testbed system). This is much better than testing the package from the source treeu during build time