Package: lintian Version: 2.105.0 Severity: normal X-Debbugs-Cc: debian...@lists.debian.org, Paul Wise <p...@debian.org>
I see lintian has recently started emitting warnings for packages that have autopkgtests, but only superficial autopkgtests. I think this is counterproductive. Obviously, if a package can have reliable autopkgtests that are not superficial (not always feasible!), then we would prefer to have those. However, if non-superficial autopkgtests are not achievable, then it's *considerably* better to have superficial autopkgtests than no coverage at all - a superficial test, like running "foo --help" and checking that it doesn't segfault or linking a trivial program to a library and checking that it can link, can at least check that the package is not *completely* broken (perhaps in time to stop a serious regression in the package or a dependency from migrating to testing). While checking the progress of the GNOME 40 transition I noticed that the maintainer of budgie-desktop has removed its superficial autopkgtest in order to silence the Lintian warning, leaving it with no autopkgtest coverage at all. I think this is the opposite of what we want! If maintainers are unable to add extensive coverage, we should be encouraging them to at least add superficial coverage. The only class of tests that should be discouraged is superficial tests that are not correctly marked with the 'superficial' keyword - but those are probably not something that is feasible to machine-detect, except in extremely simple cases, since the test is an arbitrary Turing-complete script. Some packages in the three possible categories, in order from least to most desirable: * gnome-shell-extension-caffeine is not really feasible to test at all: https://lintian.debian.org/sources/gnome-shell-extension-caffeine?version=38-1 * libsdl2-mixer only has superficial tests, which are better than nothing: https://lintian.debian.org/sources/libsdl2-mixer * dbus has actual unit tests: https://lintian.debian.org/sources/dbus I would expect that the test-related tags in these three packages should reflect that order from least to most desirable, with tag severity decreasing as we progress through the list. Something like this: * gnome-shell-extension-caffeine: a low-severity tag encouraging the maintainer to add tests if feasible, but not creating so much noise that it masks real problems (this used to be "info" severity, and I think that was entirely appropriate) * libsdl2-mixer: a low-severity tag encouraging the maintainer to add non-superficial tests if feasible, but not creating so much noise that it masks real problems (I would say "info" or "pedantic" severity) * dbus: no tags In fact they are: * gnome-shell-extension-caffeine: no tags * libsdl2-mixer: W: superficial-tests * dbus: no tags I think packages like gnome-shell-extension-caffeine used to be flagged with testsuite-autopkgtest-missing, but it looks as though the meaning of testsuite-autopkgtest-missing might have changed at some point so that it is only emitted for packages that have debian/tests/control, rather than for all packages that lack autopkgtests? smcv
