Hi All, This is a new list for the autopkgtest superficial test.
If the test done in the autopkgtest does not provide significant test coverage then it should be marked with "Restrictions: superficial". Ref: https://salsa.debian.org/ci-team/autopkgtest/-/blob/master/doc/README.package-tests.rst Examples of tests which are not significant includes (its not a complete list): 1) Executing the binary to check version Test-Command: foo -v Test-Command: foo -V Test-Command: foo --version 2) Executing the binary to check help (foo -h) Test-Command: foo -h Test-Command: foo --help 3) A Python or Perl library runs import foo or require Foo; but does not attempt to use the library beyond that. Test-Command: python3 -c "import foo" Similar to the last MBF, the intended bug text is: ************************************************************************* Subject: <package>: autopkgtest must be marked superficial Severity: important User: sudipm.mukher...@gmail.com Usertags: superficialtest It has been noticed that the autopkgtest in <package> is running a trivial command that does not provide significant test coverage: - <command being run, e.g. foo --help> Executing that command is considered to be a trivial test, which does not provide significant coverage for a package as a whole. But these tests are a useful way to detect regressions in dependencies and prevent them from breaking your package. However, it is important that we are realistic about the level of test coverage provided by these commands: most regressions cannot be detected in this way. So it is not appropriate for packages with only superficial tests to have the reduced migration time to migrate from unstable to testing as that means less opportunity for testing by users compared to the package with no tests. To support this, the keyword "Restrictions: superficial" has been defined [1]. Packages where all tests are marked with this keyword are not considered for the reduced migration age from unstable to testing, and will not be allowed to migrate automatically in later stages of the freeze [2]. Its always better to have more extensive testing than having superficial testing, which again is better than having no test. Please consider i) Adding a non-trivial test, and/or ii) Mark the trivial test with "Restrictions: superficial", similar to [3] or [4]. The Release Team has listed this issue in the list of Release Critical Issues for bullseye [5] and has mentioned that the test must be marked superficial if it is not testing one of its own installed binary packages in some way. As a result, the severity of this bug report might be increased to serious in future. [1] https://salsa.debian.org/ci-team/autopkgtest/-/blob/master/doc/README.package-tests.rst#defined-restrictions [2] https://release.debian.org/bullseye/freeze_policy.html [3] https://salsa.debian.org/utopia-team/dbus/-/commit/a80908df7d119b181eec5eb0542634a30c2ad468 [4] https://salsa.debian.org/apparmor-team/apparmor/-/commit/580667513a097088ebe579884b38ac8d8666d3b3 [5] https://release.debian.org/bullseye/rc_policy.txt ************************************************************************* Attached is the dd-list. -- Regards Sudip
dd-list
Description: Binary data