Hello, Simon McVittie [2017-01-18 12:56 +0000]: > autopkgtest currently has one level of hierarchy: a test is either an > executable script in debian/tests/ named in debian/tests/control, or a > command in debian/tests/control. > There is often a finer-grained result than that available.
Right, these are the granularity of "testbed refreshes". autopkgtest's purpose is to create ephemeral testbeds, run an opaque test in it, and copy the results back and forth, i. e. it is meant to *run* arbitrary (commonly already existing) test suites. I do *not* want autopkgtset to get into the business of becoming a framework for writing test suites in and prescribe how they must look like -- there's a ton of these available already, and every programming language has their own standard (like Perl's Test::More, Python's unittest). More related to your thought, there are also several existing reporting frameworks, such as subunit, TAP, or junit. > I would like to propose TAP as autopkgtest's protocol for finer-grained > test result reporting, something like this: TBH, I don't want autopkgtest to make that call. This already came up a few years ago, but with proposing subunit. Giving any kind of policy or recommendation towards one would make it much less useful for e. g. running test suites which already report via a different framework, or writing simple smoke-tests in 10 lines of shell (which is very common, and arguably it's not really necessary in many cases to run complex and large unittests in autopkgtest). So my thought on this is: If a test wants to provide machine-readable fine-grained results/artifacts etc., it's welcome to do that and put it into $AUTOPKGTEST_ARTIFACTS/ where it's easy to grab for a developer. Once a particular one becomes widely used in Debian, CI systems like debci could even possibly search for known ones (subunit, TAP, etc.) and present the results. But this is a disussion to be had between developers (test authors) and e. g. debci. Thanks, Martin
