Bug#905885: diffoscope: skipping tests on ci.debian.net is perhaps wrong?
Control: retitle -1 'allow to override @skip_unless_tools_exist during tests and fail the test if the tool is missing' -- cheers, Holger --- holger@(debian|reproducible-builds|layer-acht).org PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C
Bug#905885: diffoscope: skipping tests on ci.debian.net is perhaps wrong?
Control: allow to override @skip_unless_tools_exist during tests and fail the test if the tool is missing On Sun, Aug 12, 2018 at 12:03:50AM +1000, Stuart Prescott wrote: > *also* diffoscope recommends apktool but tests on ci.d.n do not fail (but > should, I think) what you say here it's half true: the autopkgtest has Depends: all the recommends that we have. What is happening is that ci.d.n/autopkgtest (no clue what piece of the puzzle is responsable for this) is downloading the test dependencies from unstable if they are not available on testing, silently. I believe you may want to get in touch with the CI team for this, I don't think there is much a package can do here. > > > Perhaps @tool_required could be extended with an "all tools required" > > > mode such that no tests are skipped or missing tools are a test failure > > > (or error). An environment variable would work for setting that mode > > > and that mode used at package build time and in autopkgtest. > > > > (One difficulty with "just" extending @tool_required is that sometimes > > we really do want to skip the tests as they require a newer/older > > version of a particular tool in order that we can correctly match up > > the output.) That's an interesting idea, and I'm retitling this bug to do this. Though what we would need to twak is the @skip_unless_tools_exist decorator we use in the tests. Also, we have comparators (and tests) for tools that Debian doesn't ship, so if we add a flag to make @skip_unless_tools_exist fail a test instead of skipping it, we would also need to provide a mechanism to "override the override" and have it really skip those :) I'm not exactly sure how we should handle the various other @skip_unless_tool_is_* and friends though... -- regards, Mattia Rizzolo GPG Key: 66AE 2B4A FCCF 3F52 DA18 4D18 4B04 3FCD B944 4540 .''`. more about me: https://mapreri.org : :' : Launchpad user: https://launchpad.net/~mapreri `. `'` Debian QA page: https://qa.debian.org/developer.php?login=mattia `- signature.asc Description: PGP signature
Bug#905885: diffoscope: skipping tests on ci.debian.net is perhaps wrong?
> Braindumping here whilst I have a second... > > > diffoscope recommends apktool but apktool is not currently in buster. > > ^^ > > Ah, this was the bit I was missing in my head. ie. not a Build-Depends :) both, actually. diffoscope FTBFS in buster because: Build-Depends: apktool , … (which is an RC bug waiting to be filed) *also* diffoscope recommends apktool but tests on ci.d.n do not fail (but should, I think) > > Perhaps @tool_required could be extended with an "all tools required" > > mode such that no tests are skipped or missing tools are a test failure > > (or error). An environment variable would work for setting that mode > > and that mode used at package build time and in autopkgtest. > > Out of curiousity, is there an environment variable that ci.debian.net > reliably exports? You'd set that environment variable in d/rules and d/tests/* (and so could other distros who want to test that packages are working) > (One difficulty with "just" extending @tool_required is that sometimes > we really do want to skip the tests as they require a newer/older > version of a particular tool in order that we can correctly match up > the output.) If the newer/older version doesn't work, surely the test should fail? cheers Stuart -- Stuart Prescotthttp://www.nanonanonano.net/ stu...@nanonanonano.net Debian Developer http://www.debian.org/ stu...@debian.org GPG fingerprint90E2 D2C1 AD14 6A1B 7EBB 891D BBC1 7EBB 1396 F2F7
Bug#905885: diffoscope: skipping tests on ci.debian.net is perhaps wrong?
Hi Stuart, > > Ah, this was the bit I was missing in my head. ie. not a Build-Depends :) > > both, actually. diffoscope FTBFS in buster [..] I'll leave these to Mattia has he has been playing about in this area recently. > You'd set that environment variable in d/rules and d/tests/* (and so could > other distros who want to test that packages are working) Sure of course one would take this approach, but just out of curiousity whether there is a 100% reliable one set by ci.debian.net? > > (One difficulty with "just" extending @tool_required is that sometimes > > we really do want to skip the tests as they require a newer/older > > version of a particular tool in order that we can correctly match up > > the output.) > > If the newer/older version doesn't work, surely the test should fail? work != differ. If it *fails*, sure (and we handle/fix those), but ironically don't guarantee that the output of diffoscope will be reproducible across (invented example here) ImageMagick 5.1 and 5.2. Regards, -- ,''`. : :' : Chris Lamb `. `'` la...@debian.org / chris-lamb.co.uk `-
Bug#905885: diffoscope: skipping tests on ci.debian.net is perhaps wrong?
Hi Stuart, Braindumping here whilst I have a second... > diffoscope recommends apktool but apktool is not currently in buster. ^^ Ah, this was the bit I was missing in my head. ie. not a Build-Depends :) > Perhaps @tool_required could be extended with an "all tools required" > mode such that no tests are skipped or missing tools are a test failure > (or error). An environment variable would work for setting that mode > and that mode used at package build time and in autopkgtest. Out of curiousity, is there an environment variable that ci.debian.net reliably exports? (One difficulty with "just" extending @tool_required is that sometimes we really do want to skip the tests as they require a newer/older version of a particular tool in order that we can correctly match up the output.) Regards, -- ,''`. : :' : Chris Lamb `. `'` la...@debian.org / chris-lamb.co.uk `-
Bug#905885: diffoscope: skipping tests on ci.debian.net is perhaps wrong?
Package: diffoscope
Version: 99
Severity: normal
Dear Maintainers,
diffoscope recommends apktool but apktool is not currently in buster. The
test suite of diffoscope currently passes without error because the tests
that use apktool are decorated with @tool_required('apktool').
Consequently:
* the code paths that use apktool are not being tested in buster
* these tests are not actually passing but ci.debian.net is saying that
the test suite has passed (skipping tests to hide errors isn't the
same as a test passing)
There is clear value in being able to skip tests for the casual user on
another platform or when not interested in diffoscope's abilities in
file formats about which one does not care. For the Debian package,
however, I think every functionality of the Debian package should be
tested in autopkgtest. The current situation does not pick up that apktool
is not installable in buster, for instance, and would not pick up
something like /usr/bin/apktool being renamed.
Perhaps @tool_required could be extended with an "all tools required"
mode such that no tests are skipped or missing tools are a test failure
(or error). An environment variable would work for setting that mode
and that mode used at package build time and in autopkgtest.
BTW other latent bugs here:
* diffoscope has an unsatisfiable recommends
* diffoscope cannot be built in buster (unsatisfiable build-deps,
this is RC)
cheers
Stuart
