Bug#932862: lintian: check for autopkgtests that do cmd --version/--help but don't have Restrictions: superficial
Control: tags -1 - moreinfo On Wed, 2019-07-24 at 11:25 -0300, Chris Lamb wrote: > Thanks for this idea and for mentioning the "superficial" Restriction; > I was not aware of that. I guess my question at this point is how you > see this interacting, if at all, with the: no-op-testsuite tag? I would say that superficial tests do provide a small amount of test coverage (command-line parsing and options processing), while no-op commands do not test the package in any way. So superficial tests do provide a small amount of value. I think that this case is distinct enough from autopkgtests testing /bin/true that it should be a separate tag, mainly so that the two issues can be of different severity and have different explanatory text. Personally I'd promote no-op-testsuite to error (and autoreject such packages) and use warning for this one. > They are, of course, reporting on different things but they > would somewhat overlap in intention and perhaps merging them might be > worth considering, or perhaps separating them out even further. I think that what we can surmise about the intent of the people creating these errors and thus the lintian info text associated with these tags is going to be different enough to warrant separation. For example no-op-testsuite is at best naivety and at worst intentionally taking advantage of the reduced testing migration delay, while the superficial tests can either be a precursor to deeper testing or just the best one can do for some packages. For example a set of packages I maintain for Discord support can only be tested by humans since testing them would require a machine to pass reCAPTCHA, register two accounts, send messages between them and then delete the accounts. Even if the first requirement were feasible, testing like this is probably constitutes abuse of the Discord service. So superficial testing is the best that can be automatically done. -- bye, pabs https://wiki.debian.org/PaulWise signature.asc Description: This is a digitally signed message part
Bug#932862: lintian: check for autopkgtests that do cmd --version/--help but don't have Restrictions: superficial
tags 932862 + moreinfo thanks Hi Paul, > A number of packages run cmd --version/--help in their autopkgtests. > This test doesn't really test the functionality of the command, just of > the command-line and options parsing. Thanks for this idea and for mentioning the "superficial" Restriction; I was not aware of that. I guess my question at this point is how you see this interacting, if at all, with the: https://lintian.debian.org/tags/no-op-testsuite.html … tag? They are, of course, reporting on different things but they would somewhat overlap in intention and perhaps merging them might be worth considering, or perhaps separating them out even further. Regards, -- ,''`. : :' : Chris Lamb `. `'` la...@debian.org chris-lamb.co.uk `-
Bug#932862: lintian: check for autopkgtests that do cmd --version/--help but don't have Restrictions: superficial
Package: lintian Version: 2.16.0 Severity: wishlist A number of packages run cmd --version/--help in their autopkgtests. This test doesn't really test the functionality of the command, just of the command-line and options parsing. autopkgtest has an option called superficial for the Restrictions field that can be used to mark a test as not really exercising any real functionality. Trivial tests like running a command with the --version or --help options should be marked as superficial. Please add a lintian warning when a test runs either a command from $PATH or a test script with just a --help or --version argument but has not marked the test as superficial. There are probably a number of different ways to write a superficial that are harder to detect (like the one in uhubctl), but those will require manual review. https://codesearch.debian.net/search?q=path%3Adebian%2Ftests%2Fcontrol+--%28version%7Chelp%29 https://salsa.debian.org/ci-team/autopkgtest/blob/master/doc/README.package-tests.rst#L302 https://codesearch.debian.net/search?q=path%3Adebian%2Ftests%2F+--%28version%7Chelp%29 -- bye, pabs https://wiki.debian.org/PaulWise signature.asc Description: This is a digitally signed message part
