Bug#788659: autopkgtest: Provide option to not implicitly run undeclared tests
Control: tag -1 pending Hello again, Dominic Hargreaves [2015-06-14 0:18 +0100]: Please could adt-run have an option to exit without running any tests if the source package in question doesn't define any? Added in http://anonscm.debian.org/cgit/autopkgtest/autopkgtest.git/commit/?id=6cfd4ea08cb3d Martin -- Martin Pitt| http://www.piware.de Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org) signature.asc Description: Digital signature
Bug#788659: autopkgtest: Provide option to not implicitly run undeclared tests
On Mon, Jun 15, 2015 at 02:32:28PM +0100, Dominic Hargreaves wrote: On Mon, Jun 15, 2015 at 07:30:34AM +0200, Martin Pitt wrote: So to clarify, you want an option to disable autodep8, right? If a source package doesn't have tests and autodep8 does not generate any either, then adt-run will already report no tests in this package and exit with 8. With a perl package and --no-autodep8 it would then do the same. I thought about 'no autodep8', but the semantics aren't quite correct, since a package could declare Testsuite: autopkgtest, but rely on the autodep8 behaviour. This is in fact the common case for pkg-perl packages, as far as I can tell. FWIW, the pkg-perl packages explicitly declare Testsuite: autopkgtest-pkg-perl Not sure if that makes any difference to the plain Testsuite: autopkgtest declaration with the current adt-run implementation, but at least it could. -- Niko Tyni nt...@debian.org -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#788659: autopkgtest: Provide option to not implicitly run undeclared tests
On Mon, Jun 15, 2015 at 07:30:34AM +0200, Martin Pitt wrote: Hello Dominic, Hello, Thank you for your reply (and indeed your prompt implementation). Dominic Hargreaves [2015-06-14 0:18 +0100]: To allow test execution environments to discover packages which provide tests, their source packages should have a ``Testsuite:`` header containing ``autopkgtest`` (which is currently the only defined value). Multiple values get comma separated, as usual in control files. My expectation having read this was that adt-run would only attempt to test packages if the Testsuite header is present; however adt-run actually appears to not look for this header at all; That's correct. If you run adt-run on a source tree/dsc etc., it will only look for debian/tests/control (or autodep8). The above is just a recommendation (should) and states the reason for that. it always processes tests for a package, including automagic test suites set up by autodep8. This is probably fine in the case of invoking adt-run with a single package, but not if you want to look for regressions using adt-run; for example I ran it against all packages I'd been rebuilding against perl 5.22, and found quite a few failures which didn't appear to be related to perl 5.22 but were instead tests which just didn't work. Right, we get the same on Ubuntu CI. Over time the failures pile up, I figure it's quite difficult for the Perl team to look at/fix all these. Please could adt-run have an option to exit without running any tests if the source package in question doesn't define any? So to clarify, you want an option to disable autodep8, right? If a source package doesn't have tests and autodep8 does not generate any either, then adt-run will already report no tests in this package and exit with 8. With a perl package and --no-autodep8 it would then do the same. I thought about 'no autodep8', but the semantics aren't quite correct, since a package could declare Testsuite: autopkgtest, but rely on the autodep8 behaviour. This is in fact the common case for pkg-perl packages, as far as I can tell. So by 'declare tests', I was really talking about having Testsuite: autopkgest in the control file. Sorry if that wasn't clear. Cheers, Dominic. -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#788659: autopkgtest: Provide option to not implicitly run undeclared tests
Hello Dominic, Dominic Hargreaves [2015-06-15 22:58 +0100]: Something to satisfy my original use case where I simply run adt-run against all the packages I'm interested in, and only the ones with 'validated' tests actually run. Ah, ok -- then grep-dctrl'ing for all packages with Testsuite: autopkgtest ought to do just that, right? But it's not too much work to inspect the Testsuite header directly, I suppose, so feel free to consider this request closed - since it sounds like that goes against the design of the package. Disabling the autodep8 tests will be useful too! Then I'll let this get closed with the --no-auto-control change in the next upload. Thanks, Martin -- Martin Pitt| http://www.piware.de Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org) -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#788659: autopkgtest: Provide option to not implicitly run undeclared tests
On Mon, Jun 15, 2015 at 04:48:11PM +0300, Niko Tyni wrote: On Mon, Jun 15, 2015 at 02:32:28PM +0100, Dominic Hargreaves wrote: On Mon, Jun 15, 2015 at 07:30:34AM +0200, Martin Pitt wrote: So to clarify, you want an option to disable autodep8, right? If a source package doesn't have tests and autodep8 does not generate any either, then adt-run will already report no tests in this package and exit with 8. With a perl package and --no-autodep8 it would then do the same. I thought about 'no autodep8', but the semantics aren't quite correct, since a package could declare Testsuite: autopkgtest, but rely on the autodep8 behaviour. This is in fact the common case for pkg-perl packages, as far as I can tell. FWIW, the pkg-perl packages explicitly declare Testsuite: autopkgtest-pkg-perl The Ruby ones declare Testsuite: autopkgtest-pkg-ruby Not sure if that makes any difference to the plain Testsuite: autopkgtest declaration with the current adt-run implementation, but at least it could. It does, because autodep8 looks at the value of the Testsuite: field. AFAICT how it all works today: - debci searches for anything that matches 'autopkgtest' in the Testsuite: field, plus its configured whitelist, to see which packages have tests to be executed. - autopkgtest does not care (or didn't, until recently) about the value of the Testsuite: field; if you give it a package, it will try to run tests for it. - autodep8 uses the value of the Testsuite: field, together with other heuristics, to autogenerate a test control file -- Antonio Terceiro terce...@debian.org signature.asc Description: Digital signature
Bug#788659: autopkgtest: Provide option to not implicitly run undeclared tests
Hello Dominic, Dominic Hargreaves [2015-06-15 14:32 +0100]: I thought about 'no autodep8', but the semantics aren't quite correct, since a package could declare Testsuite: autopkgtest, but rely on the autodep8 behaviour. Right, that happens all the time with perl/ruby, as Nico and Antonio pointed out. They (should) have a different Testsuite: header in that case. So by 'declare tests', I was really talking about having Testsuite: autopkgest in the control file. Sorry if that wasn't clear. To be honest I'm confused now. So Testsuite: autopkgtest - there should be local tests in debian/tests/control. Testsuite: autopkgtest-pkg-perl - autodep8 centralized tests. With the new --no-auto-control you can disable the latter. But as said, this field doesn't concern adt-run itself at all -- you give it a package and it'll run it. It's just a hint to the higher-layer CI system which decides which tests to run at a time. So what's missing/wrong now? Thanks, Martin -- Martin Pitt| http://www.piware.de Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org) -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#788659: autopkgtest: Provide option to not implicitly run undeclared tests
Antonio Terceiro [2015-06-15 14:01 -0300]: - autopkgtest does not care (or didn't, until recently) about the value of the Testsuite: field; if you give it a package, it will try to run tests for it. That hasn't changed, isn't planned, and wouldn't make much sense for autopkgtest itself. Martin -- Martin Pitt| http://www.piware.de Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org) signature.asc Description: Digital signature
Bug#788659: autopkgtest: Provide option to not implicitly run undeclared tests
On Mon, Jun 15, 2015 at 07:16:03PM +0200, Martin Pitt wrote: Hello Dominic, Dominic Hargreaves [2015-06-15 14:32 +0100]: I thought about 'no autodep8', but the semantics aren't quite correct, since a package could declare Testsuite: autopkgtest, but rely on the autodep8 behaviour. Right, that happens all the time with perl/ruby, as Nico and Antonio pointed out. They (should) have a different Testsuite: header in that case. So by 'declare tests', I was really talking about having Testsuite: autopkgest in the control file. Sorry if that wasn't clear. To be honest I'm confused now. So Testsuite: autopkgtest - there should be local tests in debian/tests/control. Testsuite: autopkgtest-pkg-perl - autodep8 centralized tests. With the new --no-auto-control you can disable the latter. But as said, this field doesn't concern adt-run itself at all -- you give it a package and it'll run it. It's just a hint to the higher-layer CI system which decides which tests to run at a time. So what's missing/wrong now? Something to satisfy my original use case where I simply run adt-run against all the packages I'm interested in, and only the ones with 'validated' tests actually run. But it's not too much work to inspect the Testsuite header directly, I suppose, so feel free to consider this request closed - since it sounds like that goes against the design of the package. Disabling the autodep8 tests will be useful too! Thanks! Dominic. -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#788659: autopkgtest: Provide option to not implicitly run undeclared tests
Hello Dominic, Dominic Hargreaves [2015-06-14 0:18 +0100]: To allow test execution environments to discover packages which provide tests, their source packages should have a ``Testsuite:`` header containing ``autopkgtest`` (which is currently the only defined value). Multiple values get comma separated, as usual in control files. My expectation having read this was that adt-run would only attempt to test packages if the Testsuite header is present; however adt-run actually appears to not look for this header at all; That's correct. If you run adt-run on a source tree/dsc etc., it will only look for debian/tests/control (or autodep8). The above is just a recommendation (should) and states the reason for that. it always processes tests for a package, including automagic test suites set up by autodep8. This is probably fine in the case of invoking adt-run with a single package, but not if you want to look for regressions using adt-run; for example I ran it against all packages I'd been rebuilding against perl 5.22, and found quite a few failures which didn't appear to be related to perl 5.22 but were instead tests which just didn't work. Right, we get the same on Ubuntu CI. Over time the failures pile up, I figure it's quite difficult for the Perl team to look at/fix all these. Please could adt-run have an option to exit without running any tests if the source package in question doesn't define any? So to clarify, you want an option to disable autodep8, right? If a source package doesn't have tests and autodep8 does not generate any either, then adt-run will already report no tests in this package and exit with 8. With a perl package and --no-autodep8 it would then do the same. Thanks, Martin -- Martin Pitt| http://www.piware.de Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org) -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#788659: autopkgtest: Provide option to not implicitly run undeclared tests
Package: autopkgtest Version: 3.14.3 Severity: wishlist README.package-tests.rst.gz says: To allow test execution environments to discover packages which provide tests, their source packages should have a ``Testsuite:`` header containing ``autopkgtest`` (which is currently the only defined value). Multiple values get comma separated, as usual in control files. My expectation having read this was that adt-run would only attempt to test packages if the Testsuite header is present; however adt-run actually appears to not look for this header at all; it always processes tests for a package, including automagic test suites set up by autodep8. This is probably fine in the case of invoking adt-run with a single package, but not if you want to look for regressions using adt-run; for example I ran it against all packages I'd been rebuilding against perl 5.22, and found quite a few failures which didn't appear to be related to perl 5.22 but were instead tests which just didn't work. Please could adt-run have an option to exit without running any tests if the source package in question doesn't define any? I think this would be useful in other QA contexts too. I would say that this should be the default, but that is probably too difficult to change now. Cheers, Dominic. -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
