Your message dated Mon, 21 Jul 2014 07:48:28 +0000
with message-id <[email protected]>
and subject line Bug#741330: fixed in autopkgtest 3.2
has caused the Debian Bug report #741330,
regarding autopkgtest: please add ability to wrap a script/runner/adverb around
existing tests
to be marked as done.
This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.
(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact [email protected]
immediately.)
--
741330: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=741330
Debian Bug Tracking System
Contact [email protected] with problems
--- Begin Message ---
Package: autopkgtest
Version: 2.9.1
Severity: wishlist
autopkgtest is not the only implementation of "as-installed" package testing.
Other options include:
* GNOME InstalledTests (a test is a .desktop-style file in a subdirectory
of /usr/share/installed-tests,
<https://wiki.gnome.org/Initiatives/GnomeGoals/InstalledTests>)
* Autotools "make installcheck"
* NetBSD kyua <http://code.google.com/p/kyua/wiki/About>
* Perl-style TAP tests
* whatever other ad-hoc frameworks people have come up with,
e.g. /usr/lib/telepathy-gabble-tests/twisted/run-test.sh vaguely mimics
Autotools output and was designed to be screen-scraped by Linaro's LAVA
It would be nice to be able to wrap tests designed for one of these other
frameworks to make them conform to autopkgtest syntax and requirements.
For instance, some tests are executable as-is, but should be wrapped
in "adverb" commands like env, xvfb-run and/or dbus-run-session; and the .test
files representing GNOME installed-tests are not directly executable at all,
but can be run by wrapping them in gnome-desktop-testing-runner.
autopkgtest currently requires that each test is a separate
executable in debian/tests (or some other single directory in the source
tarball). In practice, this results in the entire upstream test-suite
becoming a single autopkgtest, for instance
<http://anonscm.debian.org/viewvc/pkg-gnome/desktop/experimental/glib2.0/debian/tests/installed-tests?revision=40498&view=markup>,
making it impossible for a generic autopkgtest-based framework to track
failures over time: instead of "tests 'foo' and 'bar' have always passed,
but test 'baz' regressed between versions 1.2 and 1.3" the best available
report would be "the only test is 'installed-tests' and it regressed between
1.2 and 1.3".
Straw-man syntax proposal:
If the Interpreter for a stanza is specified, the autopkgtest
implementation will look for it in the Tests-Directory (defaulting to
debian/tests as usual). If it is found there, it will be made executable
and executed once for each test-case listed in Tests, passing the
test-case's name as an argument.
If the Interpreter does not exist as a file in the Tests-Directory,
it is taken to be a shell command-line to which the
test-case's name will be appended; commands which can typically be
used in this way include env, dbus-run-session and xvfb-run.
If an Interpreter is specified, the Tests are not required to exist
as files in debian/tests, and may contain "/" in addition to the
characters allowed in Debian package names.
Example: wrapping non-executable tests in commands from $PATH
Tests:
glib/1bit-mutex.test
glib/642026-ec.test
...
glib/vfs.test
glib/volumemonitor.test
Interpreter: xvfb-run gnome-desktop-testing-runner
Depends: gnome-desktop-testing, xvfb, ...
Example: wrapping tests that need a special environment
# debian/tests/control:
Tests: fred bill bongo
Interpreter: wrap-test.sh
Depends: dbus, dpkg-dev, xvfb, ...
# debian/tests/wrap-test.sh:
#!/bin/sh
exec dbus-run-session -- xvfb-run -- env TEST_OPTIONS=all,verbose -- \
debian/tests/"$1"
It would also be nice if the Tests could be the output of a script, or
listed in a file that may be created by the package build, or something,
so that the list of test cases doesn't have to be hard-coded in
debian/tests/control. That way, GNOME packages could use something like
gnome-desktop-testing-runner --list glib/ | sed -e 's/ (.*)$//' | LC_ALL
sort -u
to list all their test-cases automatically.
Regards,
S
-- System Information:
Debian Release: jessie/sid
APT prefers proposed-updates
APT policy: (500, 'proposed-updates'), (500, 'unstable'), (500, 'testing'),
(500, 'stable'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386
Kernel: Linux 3.13-1-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_GB.utf8, LC_CTYPE=en_GB.utf8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Versions of packages autopkgtest depends on:
ii apt-utils 0.9.15.5+b1
ii debhelper 9.20140228
ii pbuilder 0.215
ii python 2.7.5-5
ii python-debian 0.1.21+nmu2
autopkgtest recommends no packages.
Versions of packages autopkgtest suggests:
pn autopkgtest-xenlvm <none>
ii lxc 0.9.0~alpha3-2+deb8u1
ii qemu-system 1.7.0+dfsg-3
ii qemu-utils 1.7.0+dfsg-3
ii schroot 1.6.8-1
-- no debconf information
--- End Message ---
--- Begin Message ---
Source: autopkgtest
Source-Version: 3.2
We believe that the bug you reported is fixed in the latest version of
autopkgtest, which is due to be installed in the Debian FTP archive.
A summary of the changes between this version and the previous one is
attached.
Thank you for reporting the bug, which will now be closed. If you
have further comments please address them to [email protected],
and the maintainer will reopen the bug report if appropriate.
Debian distribution maintenance software
pp.
Martin Pitt <[email protected]> (supplier of updated autopkgtest package)
(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing [email protected])
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Format: 1.8
Date: Mon, 21 Jul 2014 09:35:17 +0200
Source: autopkgtest
Binary: autopkgtest
Architecture: source all
Version: 3.2
Distribution: unstable
Urgency: medium
Maintainer: Autopkgtest team <[email protected]>
Changed-By: Martin Pitt <[email protected]>
Description:
autopkgtest - automatic as-installed testing for Debian packages
Closes: 741330 749241
Changes:
autopkgtest (3.2) unstable; urgency=medium
.
Improvements:
* Add --copy option to copy a file or directory into the testbed.
* Support specifying a click name like "com.example.myapp" with --click to
test an already installed click instead of a local *.click file.
(LP: #1338899)
* Introduce "autopilot_module" click test control field for specifying an
autopilot test module name. This allows tests to require additional test
dependencies, restrictions, etc. without having to specify the full
invocation of autopilot. (LP: #1338911)
* In "unpack into temp dir" mode, ignore Python 2.x test dependencies. They
are not exposed anyway as with $PYTHONPATH we can only serve one major
Python version, and we want Python 3.
* Port remaining runners from obsolete optparse to modern argparse. This
gives a consistent command line help and support for reading options from
a file with '@'.
* Port adt-run to argparse, and clean up the CLI:
- Drop rarely used and confusing options: --instantiate, --sources-tests,
--sources-no-tests, --built-binaries-filter,
--binaries{,-forbuilds,-fortests}. Add a single new --built-binaries as
the opposite of -B/--no-built-binaries.
- CLI parsing is now done in lib/adt_run_args.py, with separate unit
tests.
- This now allows any part of the command line (for adt-run itself or the
virt-server or both) to be read from a file, using the '@' prefix.
(Closes: #749241)
- Separate the options into different logical groups, in both --help and
the manpage.
* Make --override-control work for click manifests. (LP: #1342818)
* Introduce "Test-Command:" debian/tests/control field to run a shell
command instead of a "Tests" named executable. This avoids having to write
trivial one-liner scripts or allows wrapping calls to test executables.
(Closes: #741330)
.
Bug fixes:
* Set $USER environment variable also for tests that run as root.
(LP: #1337802)
* Add lintian override for ssh-setup/SKELETON, it's deliberately not
executable.
* Increase timeout for aa-clickhook workaround for LP#1337253, 100s is often
not enough.
* ssh-setup/adb: Disabling the first-time wizards sometimes fails due to adb
being brittle at that time. Retry up to 5 times.
* Optimize aa-clickhook workaround for LP #1337253 to only rebuild the
profile of the installed click app. This reduces the time from about a
minute to about a second.
* qemu, lxc: Use VirtSubproc subprocess wrappers to get timeout handling for
all commands.
* Rewrite build depends to not rely on builders being able to fall back to
alternatives (python >= 3.3 | python3-mock), to ease backporting.
* tests/pyflakes: Use pyflakes3.
* Only run pyflakes test during package build if pyflakes3 is available.
This unbreaks backports.
* Run "click info" as the same user as the test. (LP: #1342836)
* Keep <testname>-stderr output files for "allow-stderr" tests.
Checksums-Sha1:
3a6725652835bc5b96ac2274eedd9d45bf4e5c5c 1743 autopkgtest_3.2.dsc
275122be619785dbbbb5f9c6c651a9ee2cafaa08 136649 autopkgtest_3.2.tar.gz
66b7dc9ef67a1e1a0ef8886027244ae2a7ff6ce0 116238 autopkgtest_3.2_all.deb
Checksums-Sha256:
3e34010a313a3a75eb36a179cbae8f9faff75ff40f8781e6e5c826ad276391a4 1743
autopkgtest_3.2.dsc
f6bad179e5b32ccf15fdf9125bbf4f437ea8a5dde699f85d39365b1863cb81ab 136649
autopkgtest_3.2.tar.gz
8be48778f570078f46c2d316f83c4b0f6a9d20090b5852192a3851c5cbef4b09 116238
autopkgtest_3.2_all.deb
Files:
8ef73cd7525c78ea72e8c62de8032d3c 116238 devel optional autopkgtest_3.2_all.deb
753f421cb88a6e4f569057f6b6c09fb0 1743 devel optional autopkgtest_3.2.dsc
4f1d219e9b0a2188607b21ba56c4fe15 136649 devel optional autopkgtest_3.2.tar.gz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iQIcBAEBCAAGBQJTzMNNAAoJENFO8V2v4RNH0ZYQAMU+US6nmGtakBq7k7Grsj1R
YGGenU692NGs5mL8Y6hiASHAHPEmEIGIzt5/avcGmS4xfqh9rMLpnxWRAuxuSrnH
pvwPNSN3dKZg5I4JvjCtfTY+GHY0gylzIIKfqNHclqax2AnfYq/IYjn4kJtzm8kw
8RNIpQISH90csuG20jPDLe4zfg1pRSz5J6MDNLiI3Jy+0GajpXdeMAEhFwKj66sK
j1BgpsRmGMXAiZ6W7Qhp49T4Fh/siTLt70mEbzX4jPtAxWf5Ugf4M3xR2yZFKVGs
di8T4tCtXGxgwGGoPt1RJ7yvwq2/Kyk9vA2UzX73/z2nXY6kWrPW553ZW3U5WhLj
G3vJ4UoIGcvCWt4VVIX6xZHFzlL8/R7EG1CW6ZLhSwV8EiQP8JzYm9eBfrnDaIuG
IHR1YPHq//pbAuJlzjJg8Zu/qsH5o1K92h0XEKTfZoFBDROg2OKJZP7nfcgyzuOj
eEMig1A9dOZcKIzPccwxwfM2qRoPWlyTViv+6eMkgHEY5PgX3mJMOZhOSlr5nyhE
+9d+T7sFM6TuKE9v/RMwNM2kfiPMT9R6UeNDPVjLwg7GdysEW/uQFrcfHguFmvks
/BvUupUdaZPPJUsu3nt0qEd28RdxsmIvnkXnTThQU25fjJB4pvPAcM2YroftftF7
D70ndPm9oFL79iN8NIl0
=LuQU
-----END PGP SIGNATURE-----
--- End Message ---
