On 2022-07-27 16 h 32, Julian Gilbey wrote:
On Wed, Jul 27, 2022 at 07:45:12PM +0100, Julian Gilbey wrote:
On Wed, Jul 27, 2022 at 10:26:33AM -0400, Louis-Philippe Véronneau wrote:
The way I see it:
1. We should have a Lintian tag for packages not using the new
pybuild-autodep8 autopkgtest. It would be even better if this tag would be a
pointed hint that identified 'manually' written unit test autopkgtests that
could be replaced.
This way, you get something like:
python-foo source: not-using-pybuil-autodep8 [debian/tests/unittests]
for python packages that have old 'manually' written unit test autopkgtests
and:
python-foo source: not-using-pybuild-autodep8 [no-autopkgtest]
for python packages without any autopkgtest.
2. lintian-brush (or something else, but I think lintian-brush is the right
tool) would go over these packages to:
2.1 Add the new autodep8 autopkgtests and build the package to see if they
pass
2.2 Remove the "manual" unit test autopkgtests if 2.1 succeeds
2.3 Open a bug report if 2.1 fails
I'd be wary about 2.2 and 2.3. I have several packages where I know
that an automated test will fail; there are all sorts of weird cases
where I've had to write tests manually. I would also be quite cross
if manually crafted tests were automatically removed, especially in
cases such as Simon mentioned where they do things that that
automatically generated test does not do. Another thing I could
imagine happening is that the automated test succeeds in a trivial way
- it succeeds but doesn't actually test much because of the nature of
the package.
On the other hand, a bug report saying something like the following
seems much more reasonable: "We've tested this package using the
automated autopkgtest system and it seems to work by adding the line
'Testsuite: autopkgtest-pkg-pybuild'; please check that the automated
tests cover all of the tests of your manually written debian/tests/*
and if so, then please remove them. The autopkgtest-pkg-pybuild logs
are attached." This would give the maintainer the chance to decide
how best to proceed.
Here's another alternative to steps 2.1-2.3 based on this:
For packages which currently have manually-written autopkgtests:
2.A Try removing debian/tests and adding Testsuite:
autopkgtest-pkg-pybuild to debian/control, then building the
package and running autopkgtest.
2.B If this works, then submit a bug report to the BTS as I suggested
above.
2.C If this does not work, don't do anything more; trust that the
maintainer knew what they were doing when they wrote the manual
autopkgtests.
For packages which don't currently have manually-written
autopkgtests:
2.A' Try adding Testsuite: autopkgtest-pkg-pybuild to debian/control,
then building the package and running autopkgtest.
2.B' If this works, then either Janitor adds this line to
debian/control or submit a bug report to the BTS to recommend
this. (But we would not expect Janitor to do step 2.1', so this
might have to be a different setup, or maybe the script doing
2.A' could leave a list of packages for Janitor, or something
like that.)
2.C' If this does not work, submit a wishlist bug to the BTS to
recommend that the maintainer adds some autopkgtest tests,
either by making the autodep8 system work or by writing some
manual tests.
I'd be wary about adding lintian tags for this, though: with so many
packages not being able to use the autodep8 system (I vaguely recall
someone suggesting that a third of Python packages would not be able
to use the system), that's a lot of packages getting false positives.
In particular, for the suggested first version of
not-using-pybuild-autodep8 tag (which would probably be better named
manual-autopkgtest-could-be-pybuild-autodep8), how would lintian go
about identifying which packages fall into category 2.B and which into
2.C? The second version of the tag (better named something like
no-autopkgtest-could-use-pybuild-autodep8, but that's still not very
good) is less problematic.
This all makes a lot of sense to me and I think this is the way forward,
once the MR is merged. Thanks for the discussion.
--
⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Louis-Philippe Véronneau
⢿⡄⠘⠷⠚⠋ po...@debian.org / veronneau.org
⠈⠳⣄
OpenPGP_0xE1E5457C8BAD4113.asc
Description: OpenPGP public key
OpenPGP_signature
Description: OpenPGP digital signature
