Bug#1019742: reprotest: add a variation that sets DEB_BUILD_OPTIONS=nocheck
On Thu, Sep 15, 2022 at 12:14:05PM +0200, Philip Hands wrote: > BTW If someone can give me a hint where in the code one would define > such a variation I'll cheerfully give it a go myself, but sadly it seems > that my python is too weak to find the right spot for myself. The variations are defined in reprotest/build.py (each variation is a function) and listed in VARIATIONS. Then __init__.py will pull them in with some magic. Overall I don't consider this format particularly programmer-friendly. -- regards, Mattia Rizzolo GPG Key: 66AE 2B4A FCCF 3F52 DA18 4D18 4B04 3FCD B944 4540 .''`. More about me: https://mapreri.org : :' : Launchpad user: https://launchpad.net/~mapreri `. `'` Debian QA page: https://qa.debian.org/developer.php?login=mattia `- signature.asc Description: PGP signature
Bug#1019742: reprotest: add a variation that sets DEB_BUILD_OPTIONS=nocheck
Vagrant Cascadian writes: > On 2022-09-14, Philip Hands wrote: >> Vagrant Cascadian writes: >> but also (given that the tests will have passed during the normal build) the tests failing during the varied build seems unlikely to be identifying faults that are worth fixing, and so is just a waste of cycles. >>> >>> How do you know weather the bugs it is identifying are worth fixing? It >>> could also identify non-deterministic failures, or failures triggered by >>> specific build environment configurations... >> >> The point is that if the package is reproducible, then the fact that its >> tests fail when run in a weird environment (that may never be found in >> the wild) seems rather likely to be finding errors in the tests rather >> than errors in the program that gets shipped. > > Fair point! > >> Of course, if the package is not reproducible, the tests may well fail >> because the package ends up containing new bugs that are only present in >> the variant-built package, but then its also going to show up as >> non-reproducible, so does that really make a difference? > > True, though it may make things harder to verify reproducibility in > practice, especially if it is a fairly "normal" variation that triggers > the issue... > > > It is a balancing act... > > I guess I'd be fine with the defaults to go either way, but it would be > important to be able to enable or disable however this gets implemented. Absolutely. That's why I was requesting that it be a variation in its own right, since that should allow one to specify: --variations=-nocheck and get a normal (with checks) build in the varied scenario. BTW If someone can give me a hint where in the code one would define such a variation I'll cheerfully give it a go myself, but sadly it seems that my python is too weak to find the right spot for myself. Cheers, Phil. -- |)| Philip Hands [+44 (0)20 8530 9560] HANDS.COM Ltd. |-| http://www.hands.com/http://ftp.uk.debian.org/ |(| Hugo-Klemm-Strasse 34, 21075 Hamburg,GERMANY
Bug#1019742: reprotest: add a variation that sets DEB_BUILD_OPTIONS=nocheck
On 2022-09-14, Philip Hands wrote: > Vagrant Cascadian writes: > >>> but also >>> (given that the tests will have passed during the normal build) the tests >>> failing during the varied build seems unlikely to be identifying faults >>> that are >>> worth fixing, and so is just a waste of cycles. >> >> How do you know weather the bugs it is identifying are worth fixing? It >> could also identify non-deterministic failures, or failures triggered by >> specific build environment configurations... > > The point is that if the package is reproducible, then the fact that its > tests fail when run in a weird environment (that may never be found in > the wild) seems rather likely to be finding errors in the tests rather > than errors in the program that gets shipped. Fair point! > Of course, if the package is not reproducible, the tests may well fail > because the package ends up containing new bugs that are only present in > the variant-built package, but then its also going to show up as > non-reproducible, so does that really make a difference? True, though it may make things harder to verify reproducibility in practice, especially if it is a fairly "normal" variation that triggers the issue... It is a balancing act... I guess I'd be fine with the defaults to go either way, but it would be important to be able to enable or disable however this gets implemented. live well, vagrant signature.asc Description: PGP signature
Bug#1019742: reprotest: add a variation that sets DEB_BUILD_OPTIONS=nocheck
Vagrant Cascadian writes: >> but also >> (given that the tests will have passed during the normal build) the tests >> failing during the varied build seems unlikely to be identifying faults that >> are >> worth fixing, and so is just a waste of cycles. > > How do you know weather the bugs it is identifying are worth fixing? It > could also identify non-deterministic failures, or failures triggered by > specific build environment configurations... The point is that if the package is reproducible, then the fact that its tests fail when run in a weird environment (that may never be found in the wild) seems rather likely to be finding errors in the tests rather than errors in the program that gets shipped. Even if busybox's du really does have a bug where it miscounts the sizes of files when run under the fileordering variation, I'm not sure that breaking the ability to confirm that the package is reproducible is justified in order to find that bug. I'm afraid I've not yet managed to work out what's behind the mis-counting, but my first guess is that it's more likely to be something in the fuse system presenting the data than in du's counting of it. Of course, if the package is not reproducible, the tests may well fail because the package ends up containing new bugs that are only present in the variant-built package, but then its also going to show up as non-reproducible, so does that really make a difference? Cheers, Phil. -- |)| Philip Hands [+44 (0)20 8530 9560] HANDS.COM Ltd. |-| http://www.hands.com/http://ftp.uk.debian.org/ |(| Hugo-Klemm-Strasse 34, 21075 Hamburg,GERMANY signature.asc Description: PGP signature
Bug#1019742: reprotest: add a variation that sets DEB_BUILD_OPTIONS=nocheck
On 2022-09-14, Philip Hands wrote: > I suggest adding a 'nocheck' variation, that sets DEB_BUILD_OPTIONS=nocheck > during the build, Sounds reasonable! > and enabling it by default. Less sure... > The reason for doing so is that one could imagine that a package produces > differing results depending upon whether the tests were run or not Indeed! > but also > (given that the tests will have passed during the normal build) the tests > failing during the varied build seems unlikely to be identifying faults that > are > worth fixing, and so is just a waste of cycles. How do you know weather the bugs it is identifying are worth fixing? It could also identify non-deterministic failures, or failures triggered by specific build environment configurations... > This idea is prompted by `busybox` where the tests fail in the varied > scenario, > despite the fact that the package is reproducible. > Here they are failing: > > https://salsa.debian.org/installer-team/busybox/-/jobs/3227197 > > (among other things, du produces weird results when the `fileordering` >variation is active, claiming the 1MB directoy is 2MB so the tests fail, so >the varied package is not produced, so we don't get to see that it was >reproducible: > > https://salsa.debian.org/installer-team/busybox/-/blob/master/testsuite/du/du-m-works >) Consistantly? Then, maybe the test needs to be improved? > I found a couple of ways of making the issue go away: > > 1) disabling the 'fileordering' variation, thus: > > > https://salsa.debian.org/installer-team/busybox/-/commit/17387890c73388e1f56a6ae9fbc79783095b4e86 > > https://salsa.debian.org/installer-team/busybox/-/jobs/3233259 > > 2) telling the package to skip the tests when doing the variations: > > > https://salsa.debian.org/installer-team/busybox/-/commit/5260442e8ceea220fa36bdda169978d15108f781 > > which is setting this in the salsa-ci.yml: > SALSA_CI_REPROTEST_ARGS: > --variations=environment.variables+=DEB_BUILD_OPTIONS=nocheck > > https://salsa.debian.org/installer-team/busybox/-/jobs/3235476 > > > Option 2) is what I'm suggesting making into a default variation. > > If nothing else it will speed up testing of packages with extensive test > suits. I think it's a valuable feature, but I'm not entirely sure weather it should be default or not... live well, vagrant signature.asc Description: PGP signature
Bug#1019742: reprotest: add a variation that sets DEB_BUILD_OPTIONS=nocheck
Hi Phil, On Wed, Sep 14, 2022 at 04:00:04PM +0200, Philip Hands wrote: > I suggest adding a 'nocheck' variation, that sets DEB_BUILD_OPTIONS=nocheck > during the build, and enabling it by default. [...] > Option 2) is what I'm suggesting making into a default variation. [...] > If nothing else it will speed up testing of packages with extensive test > suits. as discussed in RL, /me likes. :) Please provide a MR including documenation updates as needed. And thank you very much for using and improving reprotest! -- cheers, Holger ⢀⣴⠾⠻⢶⣦⠀ ⣾⠁⢠⠒⠀⣿⡁ holger@(debian|reproducible-builds|layer-acht).org ⢿⡄⠘⠷⠚⠋⠀ OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C ⠈⠳⣄ Nach wieviel Einzelfällen wird ein Einzelfall zum Normalfall? (Jan Böhmermann) signature.asc Description: PGP signature
Bug#1019742: reprotest: add a variation that sets DEB_BUILD_OPTIONS=nocheck
Package: reprotest Version: 0.7.22 Severity: wishlist I suggest adding a 'nocheck' variation, that sets DEB_BUILD_OPTIONS=nocheck during the build, and enabling it by default. The reason for doing so is that one could imagine that a package produces differing results depending upon whether the tests were run or not, but also (given that the tests will have passed during the normal build) the tests failing during the varied build seems unlikely to be identifying faults that are worth fixing, and so is just a waste of cycles. This idea is prompted by `busybox` where the tests fail in the varied scenario, despite the fact that the package is reproducible. Here they are failing: https://salsa.debian.org/installer-team/busybox/-/jobs/3227197 (among other things, du produces weird results when the `fileordering` variation is active, claiming the 1MB directoy is 2MB so the tests fail, so the varied package is not produced, so we don't get to see that it was reproducible: https://salsa.debian.org/installer-team/busybox/-/blob/master/testsuite/du/du-m-works ) I found a couple of ways of making the issue go away: 1) disabling the 'fileordering' variation, thus: https://salsa.debian.org/installer-team/busybox/-/commit/17387890c73388e1f56a6ae9fbc79783095b4e86 https://salsa.debian.org/installer-team/busybox/-/jobs/3233259 2) telling the package to skip the tests when doing the variations: https://salsa.debian.org/installer-team/busybox/-/commit/5260442e8ceea220fa36bdda169978d15108f781 which is setting this in the salsa-ci.yml: SALSA_CI_REPROTEST_ARGS: --variations=environment.variables+=DEB_BUILD_OPTIONS=nocheck https://salsa.debian.org/installer-team/busybox/-/jobs/3235476 Option 2) is what I'm suggesting making into a default variation. If nothing else it will speed up testing of packages with extensive test suits. Cheers, Phil.
