Bug#1010698: stunnel4: debci testsuite fails after openssl version update.
Control: retitle -1 stunnel4: runtime check on openssl too tight? Control: user debian...@lists.debian.org Control: usertag -1 - flaky Hi, On 07-05-2022 21:40, Sebastian Andrzej Siewior wrote: On 2022-05-07 20:52:41 [+0200], Paul Gevers wrote: On 07-05-2022 18:22, Sebastian Andrzej Siewior wrote: Usertags: flaky Why do you conclude that? Normally we call something flaky if it has a reasonable amount of failures in pure testing environments (so no migration runs). I'm not seeing that for tunnel4 on amd64, nor arm64. Maybe I missunderstood. According to the tracking page, it failed everywhere: We use the flaky usertag for tests that without changes sometimes pass and sometimes fail. In this case, there have hardly been failures in the past, and there is clearly a change (a new version of openssl). So there's a clear issue, but we normally don't call this situation flaky. If your claim is true (and I trust it is), I do agree that it seems that the test (and I guess this comes from a runtime check) is too strict. Retitling accordingly. Seems my retitling command was heavily flawed, let's try again. Paul OpenPGP_signature Description: OpenPGP digital signature
Processed (with 2 errors): Re: Bug#1010698: stunnel4: debci testsuite fails after openssl version update.
Processing control commands: > retitle -1 stunnel4: runtime check on openssl too tight? Bug #1010698 [stunnel4] stunnel4: debci testsuite fails after openssl version update. Changed Bug title to 'stunnel4: runtime check on openssl too tight?' from 'stunnel4: debci testsuite fails after openssl version update.'. > user debian...@lists.debian.org Unknown command or malformed arguments to command. > usertag -1 - flaky Unknown command or malformed arguments to command. -- 1010698: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1010698 Debian Bug Tracking System Contact ow...@bugs.debian.org with problems
Bug#1010698: stunnel4: debci testsuite fails after openssl version update.
On 2022-05-07 20:52:41 [+0200], Paul Gevers wrote: > Hi Sebastian, Hi Paul, > On 07-05-2022 18:22, Sebastian Andrzej Siewior wrote: > > Usertags: flaky > > Why do you conclude that? Normally we call something flaky if it has a > reasonable amount of failures in pure testing environments (so no migration > runs). I'm not seeing that for tunnel4 on amd64, nor arm64. Maybe I missunderstood. According to the tracking page, it failed everywhere: | autopkgtest for stunnel4/3:5.63-1: | amd64: Regression ♻ (reference ♻), | arm64: Regression ♻ (reference ♻), | armhf: Regression ♻ (reference ♻), | i386: Regression ♻ (reference ♻), | ppc64el: Regression ♻ (reference ♻), | s390x: Regression ♻ (reference ♻) Does this mean flaky or is this something in case it fails _always_ and not just randomly swi-prolog/8.4.2+dfsg-2 on i386 while it passed on other arches. > > that the error is that it was compiled against one version (1.1.1n) > > and then tested against another version (1.1.1o)? > > stunnel4 -version reports: > > | Compiled with OpenSSL 1.1.1n 15 Mar 2022 > > | Running with OpenSSL 1.1.1o 3 May 2022 > > > > and it is fine, the ABI is stable. > > If your claim is true (and I trust it is), I do agree that it seems that the > test (and I guess this comes from a runtime check) is too strict. Retitling > accordingly. Oki, thanks. > Paul Sebastian
Bug#1010698: stunnel4: debci testsuite fails after openssl version update.
Control: stunnel4: runtime check on openssl too tight Hi Sebastian, On 07-05-2022 18:22, Sebastian Andrzej Siewior wrote: Usertags: flaky Why do you conclude that? Normally we call something flaky if it has a reasonable amount of failures in pure testing environments (so no migration runs). I'm not seeing that for tunnel4 on amd64, nor arm64. The debci testsuite failed for all architectures after the recent openssl upload https://ci.debian.net/data/autopkgtest/testing/amd64/s/stunnel4/21436404/log.gz Am I right to assume as per | logs/results.log:DEBUG: 2022-05-07 04:07:41,445: [get_version] Trying to obtain the version of /tmp/autopkgtest-lxc.algiqp59/downtmp/build.oOV/src/tests/../src/stunnel | logs/results.log:DEBUG: 2022-05-07 04:07:41,446: [get_version] Started `/tmp/autopkgtest-lxc.algiqp59/downtmp/build.oOV/src/tests/../src/stunnel -version` as process 1544 | logs/results.log:CRITICAL: 2022-05-07 04:07:41,449: [main] Something went wrong: Stunnel was compiled and run with various OpenSSL versions … | autopkgtest [04:07:41]: test upstream: ---] | upstream FAIL non-zero exit status 1 | autopkgtest [04:07:41]: test upstream: - - - - - - - - - - results - - - - - - - - - - | autopkgtest [04:07:41]: summary | debian-pythonPASS | upstream FAIL non-zero exit status 1 that the error is that it was compiled against one version (1.1.1n) and then tested against another version (1.1.1o)? stunnel4 -version reports: | Compiled with OpenSSL 1.1.1n 15 Mar 2022 | Running with OpenSSL 1.1.1o 3 May 2022 and it is fine, the ABI is stable. If your claim is true (and I trust it is), I do agree that it seems that the test (and I guess this comes from a runtime check) is too strict. Retitling accordingly. Paul OpenPGP_signature Description: OpenPGP digital signature
Bug#1010698: stunnel4: debci testsuite fails after openssl version update.
Package: stunnel4 Version: 3:5.63-1 Severity: serious User: debian...@lists.debian.org Usertags: flaky The debci testsuite failed for all architectures after the recent openssl upload https://ci.debian.net/data/autopkgtest/testing/amd64/s/stunnel4/21436404/log.gz Am I right to assume as per | logs/results.log:DEBUG: 2022-05-07 04:07:41,445: [get_version] Trying to obtain the version of /tmp/autopkgtest-lxc.algiqp59/downtmp/build.oOV/src/tests/../src/stunnel | logs/results.log:DEBUG: 2022-05-07 04:07:41,446: [get_version] Started `/tmp/autopkgtest-lxc.algiqp59/downtmp/build.oOV/src/tests/../src/stunnel -version` as process 1544 | logs/results.log:CRITICAL: 2022-05-07 04:07:41,449: [main] Something went wrong: Stunnel was compiled and run with various OpenSSL versions … | autopkgtest [04:07:41]: test upstream: ---] | upstream FAIL non-zero exit status 1 | autopkgtest [04:07:41]: test upstream: - - - - - - - - - - results - - - - - - - - - - | autopkgtest [04:07:41]: summary | debian-pythonPASS | upstream FAIL non-zero exit status 1 that the error is that it was compiled against one version (1.1.1n) and then tested against another version (1.1.1o)? stunnel4 -version reports: | Compiled with OpenSSL 1.1.1n 15 Mar 2022 | Running with OpenSSL 1.1.1o 3 May 2022 and it is fine, the ABI is stable. Sebastian
