Hi all, On Wed, Dec 10, 2014 at 08:08:25AM +1300, Thomi Richards wrote: > Hi everyone,
> Thanks for the replies. I'd like to summarise my understanding: > - Test authors *should* add test classes that describe the test > environment they require. However, adt-run ignores these, and there's no > infrastructure piece that reads and understands these classes, so this is > purely for future-proofing. I don't think we've defined these - I suggest > just using whatever makes sense, and we can clean these up later. > - Test suites that fail will not block package promotion, unless those > failures are regressions (i.e.- they have passed in the past). > - Test authors *must* make tests skip if they're run in an incompatible > environment (hint: if you're writing autopilot tests, the > autopilot.platform module might help here). > > Did I miss anything? > Steve - does this answer your question? Ok, so just to be clear, the tests should: - be written assuming that the adt harness will run them on the appropriate device; - set a 'Class' to hint this; - use the interfaces described in https://people.debian.org/~mpitt/autopkgtest/README.package-tests.html to trigger and resume testing across reboots; - and make these tests fail gracefully when run on jenkins today. Is that correct? I think that provides most of the guidance we need. The only thing I'm still not sure about is the "fail gracefully" part. All of the interfaces invoked by our tests are things that, under DEP8, would be declared as dependencies of the tests (AIUI), and there's no interface for querying the key bit that will make the test work (i.e., "is the recovery partition set up to upgrade us, and does this device support rebooting into recovery mode?"). So for the time being I don't think we can be very graceful. Also, I saw "touch" suggested in this thread as a class name. What semantics do we expect that to express? What I need to express is "this test needs to be run on a real device, because that's the only place we can reboot into the recovery partition". I'm not sure that's the obvious semantic of "touch"; I think we should agree on what we're using these classes for before we start declaring them, otherwise I might as well not declare the class at all in the near term since there's nothing that can consume it meaningfully. Thanks, -- Steve Langasek Give me a lever long enough and a Free OS Debian Developer to set it on, and I can move the world. Ubuntu Developer http://www.debian.org/ [email protected] [email protected]
signature.asc
Description: Digital signature
-- Mailing list: https://launchpad.net/~canonical-ci-engineering Post to : [email protected] Unsubscribe : https://launchpad.net/~canonical-ci-engineering More help : https://help.launchpad.net/ListHelp
