Re: package testing, autopkgtest, and all that
On Wed, Feb 02, 2011 at 08:55:07AM +0100, Stefano Zacchiroli wrote: So, where do we start/continue sharing the thoughts on a tentative DEP? ;) Let's see first if we have all the arguments on the table already, thanks to this thread. I'm willing to co-drive a DEP to finalize the spec, although I definitely need helping hands (hint, hint!). I'm very interested in this and willing to help pushing it forward. I sense a similar interest for Yarik. Michael -- Michael Hanke http://mih.voxindeserto.de -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20110202130656.GB30281@meiner
Re: package testing, autopkgtest, and all that
Yaroslav Halchenko writes (Re: package testing, autopkgtest, and all that): First a brief question: The source package provides a test metadata file debian/tests/control. This is a file containing zero or more RFC822-style stanzas, along these lines: Do you still have somewhere that awk package demo package which had debian/tests/ ? Currently our archive does not carry any package having debian/tests/ (unfortunately). Probably. I'll see if I can dig it out. All of the above approaches seems to separate testing materials from the actual packages. Both mago and checkbox come with user interfaces, thus enabling users to test/validate their own systems without requiring setting up any virtual environment. And they ship their tests along (there seems to be some discovery mechanisms but I haven't checked in detail yet). These tools are mostly orthogonal to and even complementary to autopkgtest. There are a whole bunch of other automatic testing utilities like xnee, etc., which solve other parts of the automatic testing problem and can be usefully combined with autopkgtest. I wasn't aware of qa-regression-testing: - qa-regression-testing: collection of unit and regression tests for various components (from kernel to xine and apache) of the systems. Is not designed for distribution to the users (yet?) That seems to want to collect the tests in one central place. Debian (and to an extent our derivatives) have some difficulty with centralised collections of information about multiple different pieces of software. Our social and technical structures are based around more-or-less freestanding packages. So it makes more sense in our context to locate tests with the package. On the other hand, Ian's autopkgtest aims to provide a unified testing framework built around packages, so that it becomes possible for us, maintainers, to equip packages with necessary tests batteries; which could be reused by others (e.g. QA teams). Yes. Unfortunately the core aspect of the current autopkgtest - relying on tests in source packages -- imho to be not the ideal solution to target both sides of the userbase (i.e. maintainers/QA vs mortals). I'm not sure I know what you mean. ,--- | Q. Why put the tests in the source package ? | | A. Other possibilities include a special .deb generated by the | source (which is a bit strange and what happens to this .deb and | will make it even harder to reuse upstream test suites), or | putting them in the .deb to be tested (definitely wrong - most | people won't want the tests and they might be very large) or | having them floating about separately somewhere (which prevents us | from sharing and exchanging tests with other parts of the Free | Software community). The source package is always available when | development is taking place. `--- Ian -- could you please unroll your arguments a bit? I still do not see why source packages are beneficial besides build-time testing (which we often do already without any additional framework) You're asking why the tests should be in source packages rather than binary packages. The underlying point you seem to be missing is that a major aim of autopkgtest is that it should be straightforward to provide autopkgtest bindings for existing upstream test suites. So, firstly, binary packages have a much bigger overhead, and more complicated restrictions, compared to adding new material to source packages: - Binary packages get entries in a large number of databases both in our central infrastructure and on users' systems - Binary packages are highly visible in ways that test infrastructure doesn't need to be - They are relatively complicated to produce - They can only be installed in particular locations - Only one version can be present on a system at once (unless even more complicated things are done) The additional complexity and effort doesn't buy us anything, and the restrictions are pointless. Secondly: very often the tests will want to use other material from the source package. Upstream packages often come with tests, or strange example programs, or other material, which is useful for testing but not useful for users, and which the upstream package does not install. So if the tests were in binary packages, often we'd have to construct a weird binary package which contained all or part of the built source tree. This would be very ugly and also bulky. And additional violence would have to be done to the upstream test suite to try to make it work when installed, rather than when run out of the source tree as it expects. This is probably going to be much harder than simply arranging for the tests to test the installed rather than built copy (if they don't already). Thirdly: having the tests in a source package makes it easier to run one version of the tests against a different version of the package. This is because we typically have different distribution
Re: package testing, autopkgtest, and all that
Stefano Zacchiroli writes (Re: package testing, autopkgtest, and all that): All that considered, I'd like to know the rationale of this initial design choice as well. In particular, it would be nice to know if anyone see disadvantages in having *also* (rather then instead) support for running tests which are not part of a source package. I hope I have answered the substantive questions in my other recent email. As for also running tests which are not part of a source package, it is very easy to wrap up some tests in a dedicated source package if that's desirable. The source package is then just a convenient container format. There is no requirement in autopkgtest that the source package containing tests generates any binary packages at all, let alone that it generates the particular things being tested. So, where do we start/continue sharing the thoughts on a tentative DEP? ;) Let's see first if we have all the arguments on the table already, thanks to this thread. I'm willing to co-drive a DEP to finalize the spec, although I definitely need helping hands (hint, hint!). One point I would like to make is that people who are now raising objections to fundamental design decisions are, I think, five and a half years too late. The design, both in principle and detail, was discussed in November 2005 on various Debian lists. Amongst other people, you, Stefano, participated. In February 2006 I reported that I had an initial implementation. I don't think going back the drawing board now is a very good idea. What we are lacking is deployment (and, sorry for my part in the lack of that). Ian. -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/19785.27134.803778.102...@chiark.greenend.org.uk
Re: package testing, autopkgtest, and all that
On Wed, 02 Feb 2011 at 14:15:07 +, Ian Jackson wrote: So if the tests were in binary packages, often we'd have to construct a weird binary package which contained all or part of the built source tree. This would be very ugly and also bulky. FWIW, Maemo does this, and it's a pain to deal with in practice. Basically nobody writes upstream tests like this, leading to a lot of strange goo in debian/rules: http://gitorious.org/gnome-essentials/dbus-glib/blobs/maemo/debian/rules I think Ian's right, this approach should be avoided. This is probably going to be much harder than simply arranging for the tests to test the installed rather than built copy (if they don't already). For those familiar with Automake or the GNU coding standards, this is the distinction between make check (take the uninstalled copy I just compiled, and test it), and make installcheck (test the installed version of this package). Automake's make distcheck also runs installcheck on the just-installed binaries, if implemented. I get the impression that implementations of installcheck are also quite rare, but telepathy-mission-control-5 is one example of a package that has it. It runs a subset of the make check tests on the installed binary. (It's a subset because some of the Mission Control tests require extra hooks in the binary, to manipulate its internal state directly - the make check tests run on a modified binary with extra D-Bus interfaces to do that, but the tests relying on this functionality are skipped in make installcheck because they wouldn't work.) Simon -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20110202150428.ga26...@reptile.pseudorandom.co.uk
Re: package testing, autopkgtest, and all that
On Wed, Feb 02, 2011 at 02:28:14PM +, Ian Jackson wrote: As for also running tests which are not part of a source package, it is very easy to wrap up some tests in a dedicated source package if that's desirable. The source package is then just a convenient container format. There is no requirement in autopkgtest that the source package containing tests generates any binary packages at all, let alone that it generates the particular things being tested. I wonder how that would look in detail. For software in our field we also need to deal with test suites that upstream intents to be ran by users, that comes as a separate download with substantial size. Would I have to wrap them into a source package that builds no binary packages? Is that possible with the current infrastructure at all? Let's see first if we have all the arguments on the table already, thanks to this thread. I'm willing to co-drive a DEP to finalize the spec, although I definitely need helping hands (hint, hint!). One point I would like to make is that people who are now raising objections to fundamental design decisions are, I think, five and a half years too late. The design, both in principle and detail, was discussed in November 2005 on various Debian lists. Amongst other people, you, Stefano, participated. In February 2006 I reported that I had an initial implementation. I don't think going back the drawing board now is a very good idea. What we are lacking is deployment (and, sorry for my part in the lack of that). I don't necessarily take the point of being 5 years too late. If everything was optimal and decided 5 years ago, why is nobody using this system? Having all the other testing approaches around clearly indicates that there is demand... Michael -- Michael Hanke http://mih.voxindeserto.de -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20110202150832.GA2014@meiner
Re: package testing, autopkgtest, and all that
On Wed, Feb 02, 2011 at 10:08:32AM -0500, Michael Hanke wrote: I don't think going back the drawing board now is a very good idea. What we are lacking is deployment (and, sorry for my part in the lack of that). I don't necessarily take the point of being 5 years too late. If everything was optimal and decided 5 years ago, why is nobody using this system? Having all the other testing approaches around clearly indicates that there is demand... Just in case: I didn't mean to sound too negative. I see the rational in having things the way they are in autopkgtest, but maybe we can also think about autopkgtest as one essential core component and a general approach to distribution-wide single- and cross-package testing. Michael -- Michael Hanke http://mih.voxindeserto.de -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20110202151604.GB2014@meiner
Re: package testing, autopkgtest, and all that
Hi Ian, thanks for the input! It does make motivation behind source packages-based testing clearer. And Simon's example is a good one ;-) As a summary: source packages-based testing often provides more convenient and upstream-friendly approach, thus it must not be excluded, echoing Stefano's comment that ideally we might want having both (source and binary packages testing). Moreover we might need a simple registry/specification on how already existing packages could expose included test batteries (so echoing back to debian/README.test): e.g. many python modules have tests included and could easily be invoked with 'nosetests module' or 'python -c import module; module.test()', so we just need a specification installed on how to run the tests (and possibly common output formats so our backend could pick them up, e.g. nosetests, py.test, ctest, etc). As such, those packages do not need separate binary package, nor source package for testing. or alternatively -- they are already 'testing' binary packages without clear specification on how to invoke the tests and collect the results. Unfortunately the core aspect of the current autopkgtest - relying on tests in source packages -- imho to be not the ideal solution to target both sides of the userbase (i.e. maintainers/QA vs mortals). I'm not sure I know what you mean. in the core -- users usually do not deal with source packages; many of them do not even have deb-src lines for apt. They do not care how things are built, but if we want them to contribute by testing their systems, the simplest approach is exposing test batteries as binary packages. Of cause, user-friendly front-end might overcome those difficulties even if tests are provided in source packages. Once again, many arguments behind source packages-based testing are sound to me, but I must disagree with many statements in the bigger overhead argument: - Binary packages get entries in a large number of databases both in our central infrastructure and on users' systems and imho it is ok -- we already have -dbg packages which are also of marginal importance to users, unless they need them, so they get installed them explicitly - Binary packages are highly visible in ways that test infrastructure doesn't need to be we (at least me and Michael) do want making them visible -- that is the point. Otherwise they would not be exercised, and be left unknown (e.g. like qa-regression-testing ;-) ) - They are relatively complicated to produce yes -- unless the opposite ;) i.e. for many packages, indeed, source packages testing seems to be more adequate. - They can only be installed in particular locations I do not see it much as a disadvantage - Only one version can be present on a system at once (unless even more complicated things are done) for the goal of testing current system setup, installing the single, most recent battery, sounds sufficient. To complement there are snapshot.debian.org and backports.debian.org, so any previous or backported version could be made available -- =--= Keep in touch www.onerussian.com Yaroslav Halchenko www.ohloh.net/accounts/yarikoptic -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20110202155039.gb13...@onerussian.com
Re: package testing, autopkgtest, and all that
Michael Hanke writes (Re: package testing, autopkgtest, and all that): On Wed, Feb 02, 2011 at 02:28:14PM +, Ian Jackson wrote: As for also running tests which are not part of a source package, it is very easy to wrap up some tests in a dedicated source package if that's desirable. The source package is then just a convenient container format. There is no requirement in autopkgtest that the source package containing tests generates any binary packages at all, let alone that it generates the particular things being tested. I wonder how that would look in detail. For software in our field we also need to deal with test suites that upstream intents to be ran by users, that comes as a separate download with substantial size. Would I have to wrap them into a source package that builds no binary packages? Is that possible with the current infrastructure at all? I don't know how possible it is with the current Debian archive but I guess you could have it generate a dummy package with priority extra. Ian. -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/19785.32053.211813.590...@chiark.greenend.org.uk
Re: package testing, autopkgtest, and all that
Yaroslav Halchenko writes (Re: package testing, autopkgtest, and all that): in the core -- users usually do not deal with source packages; many of them do not even have deb-src lines for apt. They do not care how things are built, but if we want them to contribute by testing their systems, the simplest approach is exposing test batteries as binary packages. Of cause, user-friendly front-end might overcome those difficulties even if tests are provided in source packages. I don't understand what your usage model is. Are you expecting random users to execute the tests ? Why would they do that ? What kind of useful outcome is this likely to produce ? I think that someone who doesn't have a deb-src line in their sources.list, and has no knowledge of the existence of source packages, is very unlikely to produce a useful bug report which leads to an improvement to the software. Making test suites highly end-user-visible is simply likely to result in a lot of noise. - Binary packages get entries in a large number of databases both in our central infrastructure and on users' systems and imho it is ok -- we already have -dbg packages which are also of marginal importance to users, unless they need them, so they get installed them explicitly A much larger proportion of the users of libfoo-dev are likely to want to install libfoo-dbg, than the proportion of the users of coreutils who want to run its regression tests. Furthermore, if you wanted libfoo-dbg then a copy of the debug library in the built source tree is no good to you because you want it to be picked up by your existing build and runtime system for whatever you are compiling or running that links against libfoo. for the goal of testing current system setup, installing the single, most recent battery, sounds sufficient. To complement there are snapshot.debian.org and backports.debian.org, so any previous or backported version could be made available Our mechanisms for downloading and installing specific binary packages from different source are not very well-developed. Ian. -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/19785.33457.397049.476...@chiark.greenend.org.uk
Re: package testing, autopkgtest, and all that
On Wed, Feb 02, 2011 at 04:13:37PM +, Ian Jackson wrote: Yaroslav Halchenko writes (Re: package testing, autopkgtest, and all that): in the core -- users usually do not deal with source packages; many of them do not even have deb-src lines for apt. They do not care how things are built, but if we want them to contribute by testing their systems, the simplest approach is exposing test batteries as binary packages. Of cause, user-friendly front-end might overcome those difficulties even if tests are provided in source packages. I don't understand what your usage model is. Are you expecting random users to execute the tests ? Why would they do that ? What kind of useful outcome is this likely to produce ? We expect any person that is interested or asked to run test to do it and be able to do it. They would do that because they need to be sure that things work as intended and already do that in many cases. A common use case are scientific analysis toolkits: We have seen them break due to a change in a shared lib in Debian -- a change that upstream doesn't see, because they link statically against ancient and approved versions. There are various labs that are very interested in verifying that random library updates don't break their systems, which can happen with any update. If random user clicks the TestMyDebianSystem button the outcome might be a passed test and everybody is happy, or a failure with a log. I think that someone who doesn't have a deb-src line in their sources.list, and has no knowledge of the existence of source packages, is very unlikely to produce a useful bug report which leads to an improvement to the software. Think about a dashboard that collects failures and passes. Even if a single user cannot produce a good report, you'll know that something is wrong if suddenly many machines report trouble. But to get many machines, we need to make it dead-simple to participate in this type of croud-testing. We can have GUI frontends to let people do specific tests, or offer backfill job configurations for compute clusters. Making test suites highly end-user-visible is simply likely to result in a lot of noise. Higher noise, but more samples -- I'm not sure what offers the best estimate of the actual status. I see the point in having less by better-quality input to package maintainers, but again, the test results do not have to go one-by-one to a human to inspect. for the goal of testing current system setup, installing the single, most recent battery, sounds sufficient. To complement there are snapshot.debian.org and backports.debian.org, so any previous or backported version could be made available Our mechanisms for downloading and installing specific binary packages from different source are not very well-developed. Maybe that is something we should work on in this context. Michael -- Michael Hanke http://mih.voxindeserto.de -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20110202170156.GA4928@meiner
Re: package testing, autopkgtest, and all that
Michael Hanke writes (Re: package testing, autopkgtest, and all that): But to get many machines, we need to make it dead-simple to participate in this type of croud-testing. We can have GUI frontends to let people do specific tests, or offer backfill job configurations for compute clusters. Making test suites highly end-user-visible is simply likely to result in a lot of noise. Higher noise, but more samples -- I'm not sure what offers the best estimate of the actual status. Maintainers don't want large amounts of low-quality information. That is very difficult to use for fixing bugs. I see the point in having less by better-quality input to package maintainers, but again, the test results do not have to go one-by-one to a human to inspect. We don't have any infrastructure for dealing with this kind of information in bulk. I think that what you are proposing is a different kind of thing to autopkgtest and it would be best for us to deploy autopkgtest now as it already exists. There are various labs that are very interested in verifying that random library updates don't break their systems, which can happen with any update. This is easily done with autopkgtest; the only difference from your proposal is that the source package needs to be downloaded. Doing so is not difficult or troublesome, and can be done automatically. Ian. -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/19785.37041.123734.4...@chiark.greenend.org.uk
Re: package testing, autopkgtest, and all that
On Wed, Feb 02, 2011 at 05:13:21PM +, Ian Jackson wrote: Michael Hanke writes (Re: package testing, autopkgtest, and all that): I see the point in having less by better-quality input to package maintainers, but again, the test results do not have to go one-by-one to a human to inspect. We don't have any infrastructure for dealing with this kind of information in bulk. I think that what you are proposing is a different kind of thing to autopkgtest and it would be best for us to deploy autopkgtest now as it already exists. This being the second missing piece of infrastructure that is mentioned in this thread, and IMHO worth thinking about. There are various labs that are very interested in verifying that random library updates don't break their systems, which can happen with any update. This is easily done with autopkgtest; the only difference from your proposal is that the source package needs to be downloaded. Doing so is not difficult or troublesome, and can be done automatically. Fair enough. In the context of a DEP the question remains whether we want to hammer it in stone that separately distributed upstream testsuites need to be source packaged and build dummy binary packages? Moreover, I'm begining to wonder what the scope of this DEP would be: any type of testing done in Debian, or a limited subset like per-package unit/regression tests? Michael -- Michael Hanke http://mih.voxindeserto.de -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20110202174726.GA6607@meiner
Re: package testing, autopkgtest, and all that
On ke, 2011-02-02 at 17:13 +, Ian Jackson wrote: This is easily done with autopkgtest; the only difference from your proposal is that the source package needs to be downloaded. Doing so is not difficult or troublesome, and can be done automatically. I concur. However, looking things from a slightly different angle, I can see several use cases for automated testing in Debian: * the package maintainers want to know that their package works * upstream wants to know we haven't broken anything in Debian * release managers (and everyone else!) want to know the disto in general is ready to release, and also that any one updated package doesn't break other stuff (e.g., libpng might break web browsers using it) * security team wants to know the security update doesn't break anything * sysadmins want to know upgrading to a new release doesn't break their servers * I want to know I can upgrade to today's version of testing without anything major breaking Obviously, no amount of testing will be able to find all breakage, but even a relatively low test coverage can be extremely helpful, if it is targeted suitably. autopkgtest seems to me to target package maintainers, release managers, and the security team, from the list above. A different toolset and possibly testset might be necessary for other use cases. Once we have those tools, we can use them, but for now concentrating on autopkgtest is probably the best idea. -- Blog/wiki/website hosting with ikiwiki (free for free software): http://www.branchable.com/ -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/1296668875.8434.10.ca...@havelock.lan
Re: package testing, autopkgtest, and all that
On Wed, Feb 02, 2011 at 02:28:14PM +, Ian Jackson wrote:
One point I would like to make is that people who are now raising
objections to fundamental design decisions are, I think, five and a
half years too late.
The design, both in principle and detail, was discussed in November
2005 on various Debian lists. Amongst other people, you, Stefano,
participated. In February 2006 I reported that I had an initial
implementation.
I don't think going back the drawing board now is a very good idea.
What we are lacking is deployment (and, sorry for my part in the lack
of that).
I don't think anyone in this thread was suggesting to go back to the
drawing board; I surely wasn't. That is the case not because we should
have done so 5 years ago: if there is some design flaw that wasn't spot
back then---which is possible given that deployment is missing---it
should be fixed nonetheless. Rather it is the case because there is an
implementation (thanks to you) and barking around that tree without
having an alternative implementation (or a patch) to propose is
pointless.
I was merely looking for rationales that explain the interface gap
between the expectations of some participants of this thread and what is
currently available in the implementation. Looking at the arguments you
provided later on, I'm convinced that the source package wrapping is no
big overhead. My main use case of maintaining tests in some $VCS
separate from archive packages can be easily implemented by just having
a proper debian/ dir stored in that repository, which is fair enough.
Regarding the DEP (which I fully agree is just marketing, although
that kind of marketing is part of any deployment strategy in a volunteer
community), what I'm still trying to figure out are the success criteria
and the final place where the specification should land. Would policy be
a reasonable expectation, once (and if) we reach wide spread adoption?
Cheers.
--
Stefano Zacchiroli -o- PhD in Computer Science \ PostDoc @ Univ. Paris 7
zack@{upsilon.cc,pps.jussieu.fr,debian.org} -- http://upsilon.cc/zack/
Quando anche i santi ti voltano le spalle, | . |. I've fans everywhere
ti resta John Fante -- V. Capossela ...| ..: |.. -- C. Adams
signature.asc
Description: Digital signature
Re: package testing, autopkgtest, and all that
Hi Ian, First a brief question: The source package provides a test metadata file debian/tests/control. This is a file containing zero or more RFC822-style stanzas, along these lines: Do you still have somewhere that awk package demo package which had debian/tests/ ? Currently our archive does not carry any package having debian/tests/ (unfortunately). Just a brief summary for such a ignorant DDs as myself who did not know about those additional projects in Ubuntuland before they got mentioned in this thread: Ubuntu Testing Automation project has been going on for a while: https://wiki.ubuntu.com/Testing/Automation , with active (i.e. actually being developed/used) components such as - checkbox (package in Ubuntu): user-land tool (packaged and available in Ubuntu) to provide primarily hardware testing, with some basic software tests available; reporting back to launchpad.net with gathered information on hardware. - mago (package in Ubuntu): built on top of LDTP (package in Debian), provides testing of GNOME GUI. - qa-regression-testing: collection of unit and regression tests for various components (from kernel to xine and apache) of the systems. Is not designed for distribution to the users (yet?) All of the above approaches seems to separate testing materials from the actual packages. Both mago and checkbox come with user interfaces, thus enabling users to test/validate their own systems without requiring setting up any virtual environment. And they ship their tests along (there seems to be some discovery mechanisms but I haven't checked in detail yet). On the other hand, Ian's autopkgtest aims to provide a unified testing framework built around packages, so that it becomes possible for us, maintainers, to equip packages with necessary tests batteries; which could be reused by others (e.g. QA teams). So it seems to go closer toward the goals of qa-regression-testing project above, but tries to scale via making tests materials provided by the corresponding maintainers in the source packages. (As is now at least) it requires relatively complex setup of the system to start crunching the tests batteries, thus not user-oriented. For our purposes of testing scientific applications, as it was previously mentioned and exposed in the case studies on http://neuro.debian.net/proj_debtest.html page, we want the cocktail of all the above solutions ;-) with less accent on GUI testing but with convenient user-interface not requiring root access (besides possibly installation of the tests batteries). And it seems that the autopkgtest basis, maybe with some functionality from checkbox and mago (e.g. for collecting relevant software/hardware statistics and running basic system tests) could unroll into the ultimate solution for our goals. Unfortunately the core aspect of the current autopkgtest - relying on tests in source packages -- imho to be not the ideal solution to target both sides of the userbase (i.e. maintainers/QA vs mortals). https://wiki.ubuntu.com/AutomatedTesting associated with autopkgtest addresses the FAQ on why source packages: ,--- | Q. Why put the tests in the source package ? | | A. Other possibilities include a special .deb generated by the source | (which is a bit strange and what happens to this .deb and will make it even | harder to reuse upstream test suites), or putting them in the .deb to be tested | (definitely wrong - most people won't want the tests and they might be very | large) or having them floating about separately somewhere (which prevents us | from sharing and exchanging tests with other parts of the Free Software | community). The source package is always available when development is taking | place. `--- Ian -- could you please unroll your arguments a bit? I still do not see why source packages are beneficial besides build-time testing (which we often do already without any additional framework) In our view ideal solution from the user's (or maintainer/QA) point of view should involve exactly that -- interaction with binary packages since that is what everyone knows how to deal with, e.g.: sudo apt-get install apache2-tests adt-run apache2 or even, having installed X tests batteries adt-run --all # to run all installed tests batteries We could also complement it with sudo adt-install --all-relevant which would install test batteries complementing installed software packages, thus providing tailored coverage for a particular user needs. To accomplish above with tests only in source packages, would require apt-src like infrastructure (and do version/dependencies tracking on them), and moreover why should I care to keep sources on my (user's) system at all! What is relevant for testing seems to be: testing materials and already installed applications. And that is what other (checkbox/mago/qa-regression-testing) frameworks rely upon. And sure thing, for maintainers/QA it could get more evolved indeed requiring
Re: package testing, autopkgtest, and all that
On Tue, Feb 01, 2011 at 06:17:45PM -0500, Yaroslav Halchenko wrote:
Unfortunately the core aspect of the current autopkgtest - relying on
tests in source packages -- imho to be not the ideal solution to
target both sides of the userbase (i.e. maintainers/QA vs mortals).
Thanks for this related work research Yaroslav, I found it to be very
helpful for this discussion.
Ian -- could you please unroll your arguments a bit? I still do not
see why source packages are beneficial besides build-time testing
(which we often do already without any additional framework)
Not that you are explicitly saying they are not, but let me stress that
support for automated testing shipped in the source package is
useful. That way maintainers can keep them in sync with the package,
pretty much as software developers keep test suites in sync with the
code base, by committing them along side the code itself (and possibly
even doing feature commits that check in both the code and the
corresponding test at the same time).
What you are requesting, if I got it right, is support for having the
*possibility* of shipping tests elsewhere, possibly even not in a
package available in the archive. That can indeed come handy in various
scenarios, such as a separate test team with their own batteries of
tests (e.g. someone mentioned that other distros have distro-wide
regression test suites; such initiative can benefit from the extra
feature you are proposing). I also observe that the dependency
interface in the current spec is already pretty sane to handle that: if
the tests are shipped as part of the source package they can benefit
from the @ syntax, otherwise they'll need to be explicit.
All that considered, I'd like to know the rationale of this initial
design choice as well. In particular, it would be nice to know if anyone
see disadvantages in having *also* (rather then instead) support for
running tests which are not part of a source package.
So, where do we start/continue sharing the thoughts on a tentative
DEP? ;)
Let's see first if we have all the arguments on the table already,
thanks to this thread. I'm willing to co-drive a DEP to finalize the
spec, although I definitely need helping hands (hint, hint!).
Cheers.
--
Stefano Zacchiroli -o- PhD in Computer Science \ PostDoc @ Univ. Paris 7
zack@{upsilon.cc,pps.jussieu.fr,debian.org} -- http://upsilon.cc/zack/
Quando anche i santi ti voltano le spalle, | . |. I've fans everywhere
ti resta John Fante -- V. Capossela ...| ..: |.. -- C. Adams
signature.asc
Description: Digital signature
Re: package testing, autopkgtest, and all that
On Thu, Jan 27, 2011 at 02:45:57PM +, Ian Jackson wrote:
Ian: would you mind summarizing the status of that effort and
comment on whether, in your opinion, people interested on this topic
should continue from there or start over?
Sure. autopkgtest (the codebase) isn't very big but it contains
several interlocking parts. The key parts are:
Thanks a lot for this summary and for posting the specification (I've
some comments about it, that I'll post separately). The situation looks
much better than I thought; I believe it was just at an abandoned
draft stage and I wouldn't be surprised if others think the
same. Thanks for the good news!
It seems to me that, at this point, what we need is just adoption to
test drive specification and code. Also, we probably need to define some
mid/long-term success criteria like integration into policy (which as
usual will follow adoption).
dep-rant
I confess that this seems to be exactly one of those cases where the
DEP process (should) shine. By putting the specification in a more
visible place than (only) in an archive package we can give it more
visibility, easily point developers to it, monitor its acceptance
status, etc.
Is anyone interested in driving (or co-driving) a DEP about this? The
current specification looks in good shape already, so what is needed
is probably just defining criteria for acceptance and launch a final
round of RFC to fix minor glitches (if any). I'm willing to help
because I think this topic is very important for us, but I'm not
willing to do that alone.
/dep-rant
No matter the mechanism used to encode the spec, adoption will be driven
by two main ingredients: (1) communication, (2) incentive.
The need of communication is easy to establish: as this thread has
shown, very few developers were actually aware of all this valuable
work. This specification deserves an announcement on d-d-a IMHO, asking
for both comments and adoption.
The best incentive for adoption in this case is having periodic runs of
package tests, with reporting. At first glance, I'm tempted to propose
to use grid archive rebuilds to run tests. Lucas: how much work would it
be to hack your rebuild scripts and infrastructure to run tests (if
available)? Lucas' approach to log digging has usually been
collaborative: once a run is available, we ask on -qa to review
logs. This is of course not as good as automatic reporting (e.g. a-la
lintian.d.o), but is a start.
Cheers.
--
Stefano Zacchiroli -o- PhD in Computer Science \ PostDoc @ Univ. Paris 7
zack@{upsilon.cc,pps.jussieu.fr,debian.org} -- http://upsilon.cc/zack/
Quando anche i santi ti voltano le spalle, | . |. I've fans everywhere
ti resta John Fante -- V. Capossela ...| ..: |.. -- C. Adams
signature.asc
Description: Digital signature
Re: package testing, autopkgtest, and all that
On Thu, Jan 27, 2011 at 02:46:40PM +, Ian Jackson wrote:
* A specification which allows a source package to declare that it
contains tests, and how those tests need to be run. This
specification was discussed extensively on debian-devel at the
time and a copy is in the autopkgtest package, but I'll follow up
this email with a copy of it.
Here it is.
Thanks. I report below a few questions/comments, essentially about
unclear wording (to me).
The tests will be performed by executing debian/tests/fred,
debian/tests/bill, etc.
snip
The cwd of each test is guaranteed to be the root of the source
package which will have been built. HOWEVER note that the tests must
test the INSTALLED version of the program. Tests may not modify the
source tree (and may not have write access to it).
I find INSTALLED version of the program to be ambiguous. It might
refer to installed in the sense of 'debian/rules install' (or, to be
more precise, 'debian/rules binary', given that 'install' is not
dictated by policy). In that case the tests will use executables
available under debian/tmp/* or the like.
Alternatively, it might refer to the currently installed *package*. In
that case the tests will use executables installed along their $PATH.
The latter case needs some setup magic to install the just built
package, but is more convenient for many software rely on hard-coded
paths and do not permit to override them at runtime. The former case as
converse pro/cons.
I cannot look at the code right now to check which is the case as I'm
offline, but the text should probably be unambiguous in that respect
anyhow.
Depends: dpkg dependcy field syntax
Declares that the specified packages must be installed for the
test to go ahead. `@' stands for the package(s) generated by the
source package containing the tests; each dependency (strictly,
or-clause, which may contain `|'s but not commas) containing `@'
is replicated once for each such binary package, with the binary
package name substituted for each `@' (but normally `@' should
occur only once and without a version restriction).
From this text alone, it's not clear to me how this work with source
packages which generate multiple binary packages. Are all binary
packages AND-ed together as a single dependency for the test?
Given that large source packages can easily generate non-co-installable
binary packages, in those cases the maintainer should cherry pick the
specific dependencies test-by-test, am I reading it right?
If no Depends field is present, `Depends: @' is assumed. Note
that the source tree's Build-Dependencies are _not_ necessarily
installed, and if you specify any Depends, no binary packages from
the source are installed unless explicitly requested.
So, to specify no test dependencies at all, Depends: is the way to
go, right? (In one of the two scenarios I've described in the beginning
this is a completely useless corner-case scenario, while in the other it
is not).
Cheers.
--
Stefano Zacchiroli -o- PhD in Computer Science \ PostDoc @ Univ. Paris 7
zack@{upsilon.cc,pps.jussieu.fr,debian.org} -- http://upsilon.cc/zack/
Quando anche i santi ti voltano le spalle, | . |. I've fans everywhere
ti resta John Fante -- V. Capossela ...| ..: |.. -- C. Adams
signature.asc
Description: Digital signature
Re: package testing, autopkgtest, and all that
Stefano Zacchiroli writes (Re: package testing, autopkgtest, and all that): I find INSTALLED version of the program to be ambiguous. It might refer to installed in the sense of 'debian/rules install' (or, to be more precise, 'debian/rules binary', given that 'install' is not dictated by policy). In that case the tests will use executables available under debian/tmp/* or the like. Alternatively, it might refer to the currently installed *package*. In that case the tests will use executables installed along their $PATH. The latter. The latter case needs some setup magic to install the just built package, but is more convenient for many software rely on hard-coded paths and do not permit to override them at runtime. The former case as converse pro/cons. No, autopkgtest will install (dpkg -i, ultimately) the relevant stuff. The tests just need to execute it. I'd be happy to take suggestions for how to improve the wording to make this clearer. Depends: dpkg dependcy field syntax Declares that the specified packages must be installed for the test to go ahead. `@' stands for the package(s) generated by the source package containing the tests; each dependency (strictly, or-clause, which may contain `|'s but not commas) containing `@' is replicated once for each such binary package, with the binary package name substituted for each `@' (but normally `@' should occur only once and without a version restriction). From this text alone, it's not clear to me how this work with source packages which generate multiple binary packages. Are all binary packages AND-ed together as a single dependency for the test? Each dependency or-clause is replicated for each binary package. Since the or-clauses are anded by dpkg, that's very like anding together all the packages. Given that large source packages can easily generate non-co-installable binary packages, in those cases the maintainer should cherry pick the specific dependencies test-by-test, am I reading it right? Yes; otherwise the test dependencies would be unsatisfiable. If no Depends field is present, `Depends: @' is assumed. Note that the source tree's Build-Dependencies are _not_ necessarily installed, and if you specify any Depends, no binary packages from the source are installed unless explicitly requested. So, to specify no test dependencies at all, Depends: is the way to go, right? (In one of the two scenarios I've described in the beginning this is a completely useless corner-case scenario, while in the other it is not). Specifying no test dependencies at all is definitely wrong, because there would be no software installed to be tested. Ian. -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/19782.45420.877853.707...@chiark.greenend.org.uk
Re: package testing, autopkgtest, and all that
Stefano Zacchiroli writes (Re: package testing, autopkgtest, and all that): dep-rant I confess that this seems to be exactly one of those cases where the DEP process (should) shine. By putting the specification in a more visible place than (only) in an archive package we can give it more visibility, easily point developers to it, monitor its acceptance status, etc. Is anyone interested in driving (or co-driving) a DEP about this? The current specification looks in good shape already, so what is needed is probably just defining criteria for acceptance and launch a final round of RFC to fix minor glitches (if any). I'm willing to help because I think this topic is very important for us, but I'm not willing to do that alone. /dep-rant I think this would be a fine idea. I'd be happy to help with this (but my skills may lie elsewhere than what is essentially a form of marketing ...) The need of communication is easy to establish: as this thread has shown, very few developers were actually aware of all this valuable work. This specification deserves an announcement on d-d-a IMHO, asking for both comments and adoption. Perhaps I should have posted to d-d-a during development. The best incentive for adoption in this case is having periodic runs of package tests, with reporting. At first glance, I'm tempted to propose to use grid archive rebuilds to run tests. Lucas: how much work would it be to hack your rebuild scripts and infrastructure to run tests (if available)? Lucas' approach to log digging has usually been collaborative: once a run is available, we ask on -qa to review logs. This is of course not as good as automatic reporting (e.g. a-la lintian.d.o), but is a start. I'd love to see autopkgtest being run on the archive again. As I say I do test automation quite a lot in my day job now so my desire to babysit another test automation system is rather less. But I'd be happy to help and advise. Ian. -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/19782.45604.121249.744...@chiark.greenend.org.uk
Re: package testing, autopkgtest, and all that
On 31/01/11 at 00:29 +0100, Stefano Zacchiroli wrote: The best incentive for adoption in this case is having periodic runs of package tests, with reporting. At first glance, I'm tempted to propose to use grid archive rebuilds to run tests. Lucas: how much work would it be to hack your rebuild scripts and infrastructure to run tests (if available)? Hi, If there's a packaged tool to run the test suite on a given package, then it's quite easy to integrate it into my infrastructure. But I clearly do not have the time to get autopkgtest's code back in shape first. Lucas' approach to log digging has usually been collaborative: once a run is available, we ask on -qa to review logs. This is of course not as good as automatic reporting (e.g. a-la lintian.d.o), but is a start. Well, it has rarely worked like that. Most of the time, I just do the log analysis + bug filing alone. That means that the tool to run the test suite must be built with filing bugs in mind: it should provide all the needed info in the logfile, so that developers can easily reproduce the failure without asking the bug reporter. - Lucas -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20110131151326.ga2...@xanadu.blop.info
Re: package testing, autopkgtest, and all that
Lucas Nussbaum writes (Re: package testing, autopkgtest, and all that): If there's a packaged tool to run the test suite on a given package, then it's quite easy to integrate it into my infrastructure. But I clearly do not have the time to get autopkgtest's code back in shape first. Yes, there is such a packaged tool, adt-run. It worked last time I tried it. The options are a bit complicated because you have to specify exactly what you want to do; the script adt-testreport-onepackage is what I was using to invoke adt-run, and it runs it in broadly speaking two ways: adt-run --source some source package ---... to check that the source builds with dependencies satisfied from the archive, and if it contains tests, run those tests with the resulting binary packages adt-run --binaries=install --binary some binary package ---... to check that the package in question is installable But another useful one would probably be: adt-run --built-binaries-filter=_ --source some source package ---... to check that the package's tests pass with binary packages currently in the archive (which almost other things tests that the package's dependencies haven't changed so as to break it) I've enclosed a copy of the formatted manpage for adt-run, below. It seems to be missing the --instantiate option, whose usage message is: instantiate testbed now (during testing phase) and install packages selected for automatic installation, even if this might apparently not be required otherwise Well, it has rarely worked like that. Most of the time, I just do the log analysis + bug filing alone. That means that the tool to run the test suite must be built with filing bugs in mind: it should provide all the needed info in the logfile, so that developers can easily reproduce the failure without asking the bug reporter. It's possible that autopkgtest could do with some work in this area; currently the relevant output might be in more than one logfile. Ian. adt-run(1) Linux Programmer's Manualadt-run(1) NAME adt-run - test an installed binary package using the package's tests SYNOPSYS adt-run options... --- virt-server [virt-server-arg...] DESCRIPTION adt-run is the program for invoking the autopkgtest package testing machinery. autopkgtest is a facility for testing binary packages, as installed on a system (such as a testbed system). The tests are those supplied in the source package. adt-run runs each test supplied by a particular package and reports the results. It drives the specified virtualisation regime as appropriate, and parses the test description metadata, and arranges for data to be copied to and from the testbed as required. adt-run should be invoked (unless options to the contrary are supplied) in the top level directory of the built source tree, on the host. The package should be installed on the testbed. PROCESSING INSTRUCTIONS --built-tree directory Specifies that tests from the built source tree directory should be run. Note that the packages that would normally be installed as a result of * in the tests' Depends field (which includes the case where the Depends field is not specified) are not installed. The caller must explicitly instruct adt-run to install any relevant packages. --source dsc Builds dsc. The resulting binaries will (by default) be used to satisfy dependencies. The tests from that built tree will also be run (by default). The ordering is significant: each --source option should precede options whose dependencies are to be sat- isfied by the binaries it produces. --unbuilt-tree directory Specifies that tests from the unbuilt source tree directory should be run. This is very similar to specifing --source except that a directory tree (which should be pristine) is sup- plied, instead of a source package. --binary deb Specifies that deb should be used. By default it will be used to satisfy dependencies, both during building and testing, but not necessarily installed. The ordering is significant, as for --source. filename Bare filename arguments are processed as if --built-tree, --source, --unbuilt-tree or --binary was specified; the nature of the argument is guessed from the form of the filename. In the case of --built-tree, either the option must be specified, or the filename must end in a slash; two slashes at the end are taken to mean --unbuilt
Re: package testing, autopkgtest, and all that
On Fri, Jan 28, 2011 at 03:19:32PM +, Ian Jackson wrote: Iustin Pop writes (Re: package testing, autopkgtest, and all that): I think even without a full archive integration, having this spec publicised a bit among developers would be useful; I know I'm looking forward to adapting my own packages. Perhaps I should be trying to get the spec referred to in useful places ? My belief is that yes, that's a good idea. But I'm a bit surprised that other people didn't chime in on this thread. regards, iustin signature.asc Description: Digital signature
Re: package testing, autopkgtest, and all that
Iustin Pop writes (Re: package testing, autopkgtest, and all that): As it seems to me, right now this is most useful for individual maintainers to declare, and run, their own tests to ensure the built packages are fine. A good start, I'd say. Yes, you can certainly use it that way. I think even without a full archive integration, having this spec publicised a bit among developers would be useful; I know I'm looking forward to adapting my own packages. Perhaps I should be trying to get the spec referred to in useful places ? Ian. -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/19778.56964.874761.328...@chiark.greenend.org.uk
Re: package testing, autopkgtest, and all that
Stefano Zacchiroli writes (package testing, autopkgtest, and all that): Regarding this specific point (tests run on packages as if they were installed), IIRC Ian Jackson worked a bit on the matter, producing some code (autopkgtest---as mentioned elsewhere in this thread) and a specification of the interaction among tests and packages. Ian: would you mind summarizing the status of that effort and comment on whether, in your opinion, people interested on this topic should continue from there or start over? Sure. autopkgtest (the codebase) isn't very big but it contains several interlocking parts. The key parts are: * A specification which allows a source package to declare that it contains tests, and how those tests need to be run. This specification was discussed extensively on debian-devel at the time and a copy is in the autopkgtest package, but I'll follow up this email with a copy of it. * Machinery to interpret those declarations, and: - build the package (if needed) - install the package(s) needed for the runtime tests - run the tests (if any) and collect the results * Some surrounding ad-hoc shell scripts and crontab code to: - select a package to test - run the test - send the results in a fairly raw form to a webserver host - make some notes about how the test went for the benefit of the selection algorithm * A standardised interface to a virtualisation/snapshot testbed, with three implementations: Xen VMs and LVM snapshots; chroot; or simply running things on the actual host. All of this seemed to work reasonably well. The 1.2.0 in the archive is essentially identical to my bzr head so all the autopkgtest code is out there. The problems are that: * Nowhere is actually automatically running these tests on an ongoing basis. I was doing so but the machine I was doing it on was causing electrical problems in our house so I stopped. * We are lacking code to publish the results in a useful format that integrates well with the package qa pages etc. * Most packages in the archive declare no tests. This is less of a show-stopper than you might imagine because in the obvious configuration even a package which declares no tests gets _built_, so we check for buildability. * Even when I was automatically running these tests, no-one was systematically looking at the results, filing bugs, etc. I didn't have time to do this. But even publishing results on the qa pages would be a good start. * Much of this code hasn't been run for a few years and may have rotted slightly. Having a specification and some code to run the tests early on in the Wheezy release cycle would be amazing, as it will enable maintainers to add tests to their packages during the expected package updates for Wheezy. Absolutely. If someone would like to set up a machine running these tests and perhaps do some of the qa.debian.org integration, I would be delighted. I'm doing a lot of test automation in my day job now so I'm somewhat less interested in this myself (rather too much of a busman's holiday) but I'd be very happy to help in any way I can. Thanks, Ian. -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/19777.34085.432704.380...@chiark.greenend.org.uk
Re: package testing, autopkgtest, and all that
I wrote: * A specification which allows a source package to declare that it contains tests, and how those tests need to be run. This specification was discussed extensively on debian-devel at the time and a copy is in the autopkgtest package, but I'll follow up this email with a copy of it. Here it is. Ian. AUTOPKGTEST - DEFINING TESTS FOR PACKAGES = This document describes how the autopkgtest tester core (the program `adt-run') interprets and executes tests found in source packages. The source package provides a test metadata file debian/tests/control. This is a file containing zero or more RFC822-style stanzas, along these lines: Tests: fred bill bongo Restrictions: needs-root breaks-computer This example defines three tests, called `fred', `bill' and `bongo'. The tests will be performed by executing debian/tests/fred, debian/tests/bill, etc. Each test program should, on success, exit with status 0 and print nothing to stderr; if a test exits nonzero, or prints to stderr, it is considered to have failed. The cwd of each test is guaranteed to be the root of the source package which will have been built. HOWEVER note that the tests must test the INSTALLED version of the program. Tests may not modify the source tree (and may not have write access to it). If the file to be executed has no execute bits set, chmod a+x is applied to it (this means that tests can be added in patches without the need for additional chmod; contrast this with debian/rules). During execution of the test, the environment variable TMPDIR will point to a directory for the execution of this particular test, which starts empty and will be deleted afterwards (so there is no need for the test to clean up files left there). Tests must declare all applicable Restrictions - see below. The fields which may appear in the RFC822-style stanza are: Tests: name-of-test [name-of-another-test ...] This field is mandatory. It names the tests which are defined by this stanza. All of the other fields in the same stanza apply to all of the named tests. Test names are separated by whitespace and should contain only characters which are legal in package names, plus `/'. Restrictions: restriction-name [another-restriction-name ...] Declares some restrictions or problems with the tests defined in this stanza. Depending on the test environment capabilities, user requests, and so on, restrictions can cause tests to be skipped or can cause the test to be run in a different manner. Tests which declare unknown restrictions will be skipped. See below for the defined restrictions. Features: feature-name [another-feature-name ...] Declares some additional capabilities or good properties of the tests defined in this stanza. Any unknown features declared will be completely ignored. See below for the defined features. Depends: dpkg dependcy field syntax Declares that the specified packages must be installed for the test to go ahead. `@' stands for the package(s) generated by the source package containing the tests; each dependency (strictly, or-clause, which may contain `|'s but not commas) containing `@' is replicated once for each such binary package, with the binary package name substituted for each `@' (but normally `@' should occur only once and without a version restriction). If no Depends field is present, `Depends: @' is assumed. Note that the source tree's Build-Dependencies are _not_ necessarily installed, and if you specify any Depends, no binary packages from the source are installed unless explicitly requested. Tests-Directory: path Replaces the path segment `debian/tests' in the filenames of the test programs with path. Ie, the tests are run by executing /path/to/built/source/tree/path/name-of-test. path must be a relative path and is interpreted starting from the root of the built source tree. This allows tests to live outside the debian/ metadata area, so that they can more palatably be shared with non-Debian distributions. Any unknown fields will cause the whole stanza to be skipped. The defined Restrictions are: rw-build-tree The test(s) needs write access to the built source tree (so it may need to be copied first). Even with this restriction, the test is not allowed to make any change to the built source tree which (i) isn't cleaned up by debian/rules clean, (ii) affects the future results of any test, or (iii) affects binary packages produced by the build tree in the future. breaks-testbed The test, when run, is liable to break the testbed system. This includes causing data loss, causing services that the machine is running to malfunction, or permanently disabling services; it does not include causing services on the machine
Re: package testing, autopkgtest, and all that
On Thu, Jan 27, 2011 at 02:45:57PM +, Ian Jackson wrote: Stefano Zacchiroli writes (package testing, autopkgtest, and all that): Regarding this specific point (tests run on packages as if they were installed), IIRC Ian Jackson worked a bit on the matter, producing some code (autopkgtest---as mentioned elsewhere in this thread) and a specification of the interaction among tests and packages. Ian: would you mind summarizing the status of that effort and comment on whether, in your opinion, people interested on this topic should continue from there or start over? Sure. autopkgtest (the codebase) isn't very big but it contains several interlocking parts. The key parts are: * A specification which allows a source package to declare that it contains tests, and how those tests need to be run. This specification was discussed extensively on debian-devel at the time and a copy is in the autopkgtest package, but I'll follow up this email with a copy of it. * Machinery to interpret those declarations, and: - build the package (if needed) - install the package(s) needed for the runtime tests - run the tests (if any) and collect the results * Some surrounding ad-hoc shell scripts and crontab code to: - select a package to test - run the test - send the results in a fairly raw form to a webserver host - make some notes about how the test went for the benefit of the selection algorithm * A standardised interface to a virtualisation/snapshot testbed, with three implementations: Xen VMs and LVM snapshots; chroot; or simply running things on the actual host. All of this seemed to work reasonably well. The 1.2.0 in the archive is essentially identical to my bzr head so all the autopkgtest code is out there. Excellent. I've read your followup email and the spec seems very good (for my purposes). The problems are that: […] As it seems to me, right now this is most useful for individual maintainers to declare, and run, their own tests to ensure the built packages are fine. A good start, I'd say. Having a specification and some code to run the tests early on in the Wheezy release cycle would be amazing, as it will enable maintainers to add tests to their packages during the expected package updates for Wheezy. Absolutely. If someone would like to set up a machine running these tests and perhaps do some of the qa.debian.org integration, I would be delighted. I think even without a full archive integration, having this spec publicised a bit among developers would be useful; I know I'm looking forward to adapting my own packages. thanks, iustin signature.asc Description: Digital signature
