Bug#1033103: superficial-tests: false positive when both Testsuite + d/t/control are used

2023-03-17 Thread Faidon Liambotis 
On Fri, Mar 17, 2023 at 01:53:00PM +0200, Faidon Liambotis wrote:
> Indeed, autodep8 generates a file that has the manual tests combined
> with a pybuild-autopkgtest one. The pybuild-autopkgtest one is *not*
> marked as as superficial[1], as evident by the autodep8 output.
>
> 1: Debatable in this particular package, but besides the main point.

In case you go looking at the package to reproduce, note that in the
final upload to experimental I added extra_restrictions=superficial to
debian/tests/autopkgtest-pkg-pybuild.conf, so the autodep8 test will now
also be marked as superficial. In this case the lintian warning is
appropriate.

The bug however stands (it's present when autopkgtest-pkg-pybuild.conf
is not present). It can be reproduced by:
  apt source esptool=4.5.1+dfsg-0.1
  cd esptool*
  rm debian/tests/autopkgtest-pkg-pybuild.conf
  autodep8  # should list a d/t/control with a non-superficial test
  dpkg-buildpackage -S
  lintian ../esptool*dsc

Faidon

Bug#1033103: superficial-tests: false positive when both Testsuite + d/t/control are used

2023-03-17 Thread Faidon Liambotis 
Source: lintian
Version: 2.116.3
Severity: normal

In the experimental branch of src:esptool, I've defined the following:
  * Testsuite: autopkgtest-pkg-pybuild in debian/control, to test the
Python module. 
  * Three superficial tests in debian/tests/control to test the CLI
tools using --help.

TTBOMK this is valid configuration. autodep8(1) has a section titled
"COMBINING AUTO-GENERATED TESTS WITH MANUALLY SPECIFIED ONES" to cover
this use case.

Indeed, autodep8 generates a file that has the manual tests combined
with a pybuild-autopkgtest one. The pybuild-autopkgtest one is *not*
marked as as superficial[1], as evident by the autodep8 output.

As another piece of evidence, autopkgtest reports:
  autopkgtest [13:39:27]:  summary
  smoke-esptoolPASS (superficial)
  smoke-espefuse   PASS (superficial)
  smoke-espsecure  PASS (superficial)
  pybuild-autopkgtest  PASS

However, lintian is warning about "superficial-tests", and says "The
source package declares tests in the debian/tests/control file but
provides only tests with a superficial restriction.".

It looks to me like a lintian bug, but let me know if I've misunderstood
something.

Regards,
Faidon

1: Debatable in this particular package, but besides the main point.